AFHRL-TR-81-44(in) 


AIR  FORCE  m 


OPERATIONAL  TEST  AND  EV  ALUATION  HANDBOOK 
FOR  AIRCREW  TRAINING  DEVICES: 
OPERATIONAL  Sl  ITABILITY  EVALl  ATION 


William  V’.  Hagtn 
Scephen  R.  Osborne 
Roik  L.  Hoekenberger 
James  P.  Smith 
Seville  Research  Corporation 
iOO  Plaza  Building 
Pensacola.  Florida  32503 


Thomas  H.  Orav 


OPERATIONS  TRAINING  DIV  ISION 
Williams  Air  Force  Base,  Arizona  85224 


Februarv  1982 


Final  Report 


Approvcd  for  public  release:  distribution  unlimited.  . 

C " 


CO 


LABORATORY 


AIR  FORCE  SYSTEMS  COMMAND 

BROOKS  AIR  FORCE  BASEJEXAS  78235 


NOTICK 

Vlhnil  A>vt*rniiii*iit  ilraHin^s,  s|M‘rifi4‘u(ions.  or  ollu*r  dalu  art'  used  tor  aiu  pur|H4st‘  oiiirr  diaii 
in  t'oniO'tiion  wilh  a  di'finili'lv  (/ovornni<‘iU-ndal<*d  j»nM  un*mt*nt.  ilu*  t  ni(i*<t  StaU*s 
(»ovorniMi‘n{  inc  urs  no  rc'sponsibilily  or  any  o)>ligalion  wliatsoc*v<‘r.  'Ftic*  fac  t  tlial  iht* 
(jovc^rnnit'nt  may  have*  forinulatcHi  or  in  any  v>ay  supplit*d  the*  saict  dravsinfjs.  spc'c  ifirations.  or 
olht*r  data,  is  not  to  he*  rt*^ardt*d  hy  implication,  or  olhc'rwisc  in  any  manm'r  construed,  as 
ficc'iising  the  liotder.  or  any  otiier  pc*rson  or  corporafiori:  cir  as  coni<*>in^  aiM  rights  or 
peniiission  to  manufacture*,  use.  t>rsc*ll  anv  patentc*d  invc*ntion  tliat  mav  in  anv  wav  he  related 
thereto. 

The*  Puhlie*  Affairs  Office  has  revie*wed  this  report,  and  it  is  releasable  to  tlee*  Natitjnal 
IVchnical  Information  Se*rvice.  whe*re*  it  will  he  available  to  the  pene*ral  puhlie.  including 
fe»reign  nationals. 

This  report  has  he*en  revie*wed  aiul  is  approve*d  for  publication. 

THOMAS  H.  (;RAY 
Contract  Monitor 

MILTON  K.  Vl  (K)l).  IVchnical  DircM  tor 
Ope*ratiems  Training  Divisiem 


RONALD  VI  .  TKRRY.  Colonel.  I  SAF 
Cotninander 


Inilassil’iiHl 


security  Cl,  ASSiFiC  ATION  of  This  page  (Whvn  ^ _ 

REPORT  DOCUMENTATION  PAGE  ^  ,km 

1.  report  NUMB^  govt  accession  Nol  3  RFC  1^1  EN  ^‘S  C  ATal  ->G  NuMUER 


11  report  number 


\KHRI.-TR-81-H(I1I) 


14  TITlE  (and  Subrtf Jt*) 


ORKR  V  riON Al.  t’KST  AM)  t:\  ALl  ATION  HANDBOOK 
rOR  AlRt.RKVl  TRAIMM;  DKVK.KS:  OPI  RATIOWI, 
SI  ITVBII  ITV  RVAl.l  ATION 


5  '^VPE  OF  REPORT  b  PERlOT^  COVERED 


6  PERFORMtNO  O  ■?  S  REF'OR^  NjVBER 


|7.  AUThOR's; 


M  illiarii  \  .  flavin 
Stephen  R.  Osborne 
Roik  I-  HtM'kenberger 


James  P.  Smith 
rhoiiias  H.  (irav 


9  performing  ORGAN!  2  ATION  NAME  AND  ADDRESS 
Seville  Resean  h  (!orporalion 
1(HI  Plu/a  Ruihhn^ 

Pensaeola.  Florida 

n.  controlling  OFFICE  name  and  address 

Air  horee  I  in  man  Rt‘soum‘s  L.ahoralory  (  \FS(’.) 
Brooks  Air  Force  Base.  Texas  782 .k”) 


contract  or  grant  Number 


F;id()ir>-:8-(:-(HK);i 


10  program  ElEmFnT  PROJEC^ 
AREA  a  i«rORK  uN'"^  NUMBERS 

()22<»:.F 

12  report  date 

Fchruar\  I 

Ti  number  of  p  ages 


'  t4.  MONITORING  AGENCY  NAME  a  ADDRESSr//  different  from  l\jnfrof I totf  Offxce 


15  security  class,  ^<r  rbrs  rer  >t 


Operations  Training  I)ivision 

Air  Fon  e  Munian  Resources  Laboratory 

Viilliams  Air  F<»rc*e  Base,  Arizoina  8r>22l- 


I  nclassifieil 


ISa  DECLASSIFICATION  DOWNGRADING 

schedule 


I  16.  distribution  statement  (of  this  Report) 


Approved  tor  publle  release:  distribution  iinliiniled. 


t7.  Distribution  statement  (of  the  «hstrart  entered  .r  Block  20,  if  different  from  Report) 


18.  supplementary  notes 


f 

'v 


I  19.  KEY  WORDS  fCorWinue  on  fei-orse  .v.'de  if  necessary  and  identifv  hv  hinck  rnin'f 


aircrew  training  deviei*s 
human  tat'lors  enginetTing 
(piest  ion  na  ires 
rating  scab's 


svstein  opt‘rabiti(\ 
study  design 
test  and  evaluation 
transfer  rd  training 


20  abstract  r^Contlnue  on  reverse  side  ff  necessary  and  identify  br  him  k  nnwi^er 

Tfie  llandbiHik  is  cornprisi'd  cd  lliree  vobiim*s  amt  i.*>  inlemJed  to  pr<u  idc  guirtebm'v  arid  procedures 
appropriate  for  Air  Force  Operational  Test  and  Kvabiatioii  (O’l  xF)  personnel  to  use  in  planning,  conducting  ami 
re|Nirtiiig  the  n'sults  of  simulator  assessment  I'fforts.  Alliiough  of  value  to  all  t<'st  personnel,  it  is  primarilv  f«tr  the 
typical  novice  test  manager/director ~a  person  who  has  subject  matter  expertise  (c.g..  a  ipialified  pilot  or 
operator),  but  who  may  have  little  or  no  previous  ()1  ^F  »*\perieiiee.  Fbe  Handbook  provides  detailed  coverage  on 
f)Tc\'K  planning  and  inanageinent  with  special  emphasis  on  measuring  device  opi*rational  cffectiv cnes>.  and 
suitability.  In  aci'ord  witfi  its  objectives,  the  llandlmok  was  prepared  to  serve  as  a  'supplement  to  \ir  force  Manual 


00  ,  1473 


EDITION  OF  1  NOV  65  IS  OBSOLETE 


(  nelassified 

security  CLASSIFICATION  OF  TmiS  PAGE  EnfSfSd* 


I  nclassifiiul 

security  classification  of  this  PAGe(WT»n  Dmtm  Enffd) 


lliMU  {(.{pnlinurti) 


!>r>- n.  ’^MiiiiiijH‘im‘nl  of  ()|)(Tatioiial  IVsl  and  KvaUialn)!!.’^  Ii\  providing  lluisi*  s|M‘(  ifir  adilitioiial  f\ 
(Uiu  rpl.s  and  l<M  hfin|ii<‘s  rn‘('<»ssar\  for  itirtTt'n  /raiiiinf?  drvin*  li*sl  and  t^valuation. 


I  nrlassifird 


aluation 


security  cl  ASSiriCATlOR  OE  J’AGCrMT,, 


PREFACE 


This  volume  (Volume  III.  Operational  Suitability  Evaluation)  is 
one  part  of  a  three-volume  Handbook  produced  for  the  U.S.  Air  Force 
Human  Resources  Laboratory/Operations  Training  Division  (AFHRL/OT). 
The  Handbook  is  entitled,  "Handbook  for  Operational  Test  and  Evalua¬ 
tion  (OTSE)  of  the  Training  Utility  of  Air  Force  Aircrew  Training 
Devices."  This  effort  has  been  accomplished  by  the  Seville  Research 
Corporation  under  Contract  No.  F3361 5-78-C-0063 .  Dr.  Thomas  H.  Gray 
served  as  the  Air  Force  Laboratory  Contract  Fbnitor  (AFLCM)  on  the 
project.  For  Seville,  Dr.  William  H.  Hagin  was  Project  Director,  and 
Dr.  Wallace  W.  Prophet  was  Program  Manager. 

The  three  volumes  which  comprise  the  total  Handbook  are  intended 
to  provide  guidelines  and  procedures  appropriate  for  use  of  Air  Force 
ATD  OT&E  test  team  personnel  in  planning,  conducting,  and  reporting  the 
results  of  aircrew  training  device  OT&E  efforts.  The  three  Handbook 
volumes  are: 

Volume  I.  Planning  and  Management 

Volume  II.  Operational  Effectiveness  Evaluation 

Volume  III.  Operational  Suitability  Evaluation 

It  is  important  that  the  reader  understand  that  this  Handbook  was 
prepared  to  serve  as  a  supplement  to  AFM  56-43,  "Management  of  Opera¬ 
tional  Test  and  Evaluation"  by  providing  those  specific  additional 
evaluation  concepts  and  techniques  necessary  for  ATD  test  and 
eval uation . 
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PURPOSE  AND  ORGANIZATION  OF  VOLUME  III 


INTRODUCTION 

Volumes  I  and  II -of  this  Handbook  provide  guidance  for  the  test 
director  concerning  how  to  plan  for  ATD  OT&Es  and  how  to  conduct  ATD 
operational  training  effectiveness  evaluations.  This  volume  is  con¬ 
cerned  with  assessing  the  impact  of  operational  suitability  factors  on 
ATD  usefulness.  Operational  suitability  factors  pertain  to  how  well 
the  device  meets  accepted  equipment  serviceability  requirements  within 
its  intended  operating  and  maintenance  environment. 

There  are  two  principal  components  of  ATD  suitability  that  must 
be  examined  during  ATD  OTXE,  viz,  hardware  suitability  and  software 
suitability.  Assessment  of  hardware  suitability  is  important  from  the 
standpoint  that  OTAE  findings  in  this  area  can  signi ficantly  affect 
downstream  ATD  life-cycle  costs,  manning  requirements,  logistics  sup¬ 
port,  and  the  ultimate  instructional  usefulness  of  the  device.  For 
example,  a  device  may  be  rated  highly  with  respect  to  operational 
effectiveness,  but  it  will  be  of  little  use  if  it  is  not  available  for 
training  because  of  its  poor  reliability  or  maintainability  charac¬ 
teristics. 

Software  suitability  assessment  also  has  become  a  key  element  in 
ATD  suitability  evaluations,  because  modern  digital  ATDs  have  a 
substantial  software  component.  Software  is  involved  to  a  large 
degree  in  the  simulation  of  dynamic  flight  characteristics  and  in  pro¬ 
cedural,  navigational,  communications,  weapons,  and  electronic 
countermeasures  (ECM)  system  functions,  among  others.  Software  also 
plays  an  important  role  in  the  control  and  functioning  of  ATD  instruc¬ 
tional  support  features  (e.g.,  freeze,  record/ repl ay ,  reset,  and 
auto-demonstration),  as  well  as  automated  maintenance  test  features. 

Management  of  ATD  Operational  Suitability  Testing 


Effective  assessment  of  ATD  operational  suitability  during  OT&E 
depends  upon  careful  advanced  planning  and  skilled  execution  of  the 
required  testing  activities.  As  will  be  seen  in  the  detailed  tech¬ 
nical  discussions  of  these  testing  activities  which  follow  in  Chapters 
2  and  3  of  this  volume,  no  single  person  is  likely  to  have  the  full 
range  of  expertise  required  either  to  plan  for  or  to  execute  all  of 
the  technical  efforts  appropriate  to  ATD  operational  suitability 
testing.  For  this  reason,  two  deputy  test  directors  are  responsible 
for  the  technical  aspects  of  ATD  operational  suitability  testing: 
One,  the  Deputy  for  Logistics  Evaluation  (DLE)  is  responsible  for 
hardware  suitability  testing;  the  other,  the  Deputy  for  Software 
Evaluation  (DSE)  is  responsible  for  software  suitability  evaluation. 
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Deputy  for  Logistics  Evaluation  (DLE).  The  Deputy  for  Logistics 
Eva! uation  {DLE)f  and  his  team  are  under  the  operational  control  of 
the  AFTEC  or  MAJCOM  test  director,  and  function  in  accordance  with  the 
pertinent  provisions  of  the  following  documents:  (1)  AFR  80-14, 
"Research  and  Development  Test  and  Evaluation;"  (2)  AFM  55-43, 
"Management  of  Operational  Test  and  Evaluation;"  (3)  AFLCR  80-4,  "Test 
and  Evaluation;"  and  (4)  AFTECP  400-1,  "Logistics  Assessment." 

The  DLE  normally  is  a  senior  OT&E  specialist  from  the  Air  Force 
Logistics  Command  (AFLC),  and  his  team  consists  of  a  select  group  of 
logisticians,  engineers,  technicians,  and  other  support  specialists  as 
required  from  the  AFLC,  AFTEC,  and/or  the  MAJCOMs.  The  DLE  and  his 
team  serve  as  the  logistics  test  focal  point  in  the  accomplishment  of 
the  actions  necessary  to  effect  all  suitability  test  planning  and  exe¬ 
cution  requirements. 

Deputy  for  Software  Evaluation  (DSE).  The  Deputy  for  Software 
Eval uation  (DSE )  and  Fns  team  al  so  are  uinder  the  operational  control 
of  the  test  director.  The  DSE  and  his  team  of  software  evaluators 
operate  in  general  accord  with  those  software  evaluation  guidelines 
and  procedures  which  have  recently  been  documented  in  a  five-volume 
AFTEC  handbook  titled,  "Software  OTAE  Guidelines."  Volumes  in  this 
handbook  set  include  the  following: 

I.  Software  Test  Manager's  Handbook 

II.  Handbook  for  Deputy  for  Software  Evaluation 

III.  Software  f^aintainabil  ity  Evaluator's  Handbook 

IV.  Software  Operator-Machine  Interface  Evaluator's  Handbook 

V.  Computer  Support  Resources  Evaluator's  Handbook 


PURPOSE  OF  THIS  VOLUME 

Even  though  the  test  director  is  dependent  upon  these  two  depu¬ 
ties  (the  DLE  and  the  DSE)  for  the  technical  adequacy  of  ATD  opera¬ 
tional  suitability  testing,  he  must  have  a  general  understanding  of 
the  processes  involved.  This  volume  is  intended  to  provide  him  with 
that  understanding.  Familiarity  with  the  content  of  Volume  III  will 
help  the  test  director  by  facilitating  his  interactions  with  the  two 
suitability  test  deputies  and  their  teams  of  specialists. 


The  term  "Logistics  Support  Evaluation  Team  (LSET)"  has  been 
used  in  many  earlier  OT&E  plans  and  reports.  The  current  term  "DLE" 
has  replaced  "LSET  Chairman,"  but  the  same  functions  are  performed. 
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ORGANIZATION  AND  CONTENT 


This  Handbook  volume  consists  of  three  chapters  and  a  number  of 
appendices.  This  introductory  chapter  has  emphasized  the  importance 
of  ATD  suitability  testing.  It  has  also  pointed  out  that  the  tech¬ 
nical  complexity  of  hardware  and  software  suitability  testing  necessi¬ 
tates  the  designation  of  two  deputy  test  directors- -one  for  each  of 
the  two  areas.  Chapter  1  concludes  by  emphasizing  that  the  test 
director  must  have  a  basic  familiarity  with  the  details  of  hardware 
and  software  suitability  testing,  even  though  he  depends  upon  his  two 
deputies  (the  DLE  and  the  DSE)  for  their  actual  accomplishment. 

Chapters  2  and  3  provide  the  technical  and  methodological  infor¬ 
mation  that  the  test  director  needs  to  understand  and  manage  hardware 
and  software  suitability  testing.  Chapter  2  addresses  hardware  suita¬ 
bility  testing  in  terms  of  device  reliability,  maintainabil ity, 
availability,  logistics  supportabil  ity ,  and  operating  and  support 
costs.  Chapter  3  discusses  software  suitability  evaluation  in  terms 
of  software  maintainability  and  usability. 

The  discussions  of  each  of  the  above  listed  sub-elements  of  hard¬ 
ware  and  software  evaluation  provide  the  test  director  with  the  nec¬ 
essary  ATD  OTAE  oriented  definitions  of  these  factors  and  a  required 
understanding  of  the  methods  used  for  their  evaluation.  Perhaps  of 
even  greater  significance  to  the  test  director  are  those  parts  of  the 
discussion  which  alert  him  to  a  number  of  specific  concerns  to  which 
he  should  attend  in  the  areas  of  "Phase  of  Test  Considerations"  and 
"Personnel  Requirements." 

Phase  of  Test  Considerations 


Estimates  based  upon  OT&E  data  drive  a  number  of  critical  deci¬ 
sions  relating  to  funding/budget  planning,  manpower  requirements,  and 
planning  for  overall  support  of  the  system  in  its  operational  setting. 
The  precision  of  those  estimates  is  a  matter  of  great  concern  to  deci¬ 
sion  makers  with  respect  to  both  operational  effectiveness  and  opera¬ 
tional  suitability  factors,  because  long-term  budgetary,  manpower  and 
provisioning  plans  must  often  be  made  well  in  advance  of  actual  need. 

As  shov/n  in  Figure  1-1,  early  in-plant  I0T4E  provides  "rough" 
data  relating  to  system  effectiveness  and  suitability,  data  which  can 
then  be  refined  by  later  on-site  lOT&E  and  FOT&E  phases.  Because  the 
iterative  nature  of  this  process  of  "estimate,  refine,  and  re-evaluate" 
is  of  particular  importance  to  suitability  considerations  as  testing 
progresses,  in  the  discussions  of  the  various  suitability  factors 
which  follow,  the  section  "Phase  of  Test  Considerations"  will  include 
additional  information  that  may  be  of  use  to  the  test  director  with 
respect  to  the  specific  evaluation  element  in  question  and  the  phase 
of  test  being  supported. 
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1-1.  Precision  of  suitability  estimates 
for  successive  phases  of  OT&E. 
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Personnel  Requirements 


As  noted  above,  the  test  director  depends  upon  his  two  deputies 
for  support  in  planning,  executing,  and  reporting  ATD  suitability 
evaluation  activities.  In  most  instances,  personnel  needs  will  be 
taken  care  of  for  him  by  these  two  personnel.  However,  there  can  be 
specific  personnel  requirements  and/or  characteristics  of  direct 
interest  to  him.  Where  appropriate,  such  personnel  requirement  con¬ 
cerns  are  surfaced  in  the  concluding  portion  of  each  discussion. 
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CHAPTER  2 


HARDWARE  SUITABILITY  TESTIMG 


COMPOSITION  OF  LOGISTICS  EVALUATION  TEAM 

As  was  pointed  out  in  Chapter  1,  responsibility  for  hardware 
suitability  testing  is  assigned  to  the  Deputy  for  Logistics  Evaluation 
(DLE)  and  his  team  of  specialists.  The  composition  and  command 
sources  of  the  DLE  team  may  vary  with  each  individual  test  situation, 
but  the  following  is  illustrative  of  the  usual  team  membership: 

Posi tion  Command  source 

Deputy  Director  for  Logistics 

Evaluation  AFLC 

Reliability  and  maintainability 
engineers  (Note:  In  some  cases, 
a  simulator  maintenance  analyst 


may  serve  this  function)  AFLC 

Simulator  technicians  MAJCOM 

Technical  data  specialists  AFLC 

Flight  crews  and/or  instructors  MAJCOM 


(Note:  Resources,  normally  available  because  they  are  necessary 
for  operational  effectiveness  evaluations,  are  used  to  exercise  the 
ATD  during  suitability  testing.) 

Specific  DLE  Responsibilities 


The  Deputy  for  Logistics  Evaluation  is  responsible  for  the 
following: 

•  Assisting  the  AFTEC  or  MAJCOM  test  director  in  preparing  test 
plans  and  reports. 

•  Maintaining  open  communications  through  AFTEC  with  the  ATD 
program  office  (SimSPO),  MAJCOM,  and  AFLC. 

•  Assuring  that  written  procedures  are  available  for  data 
collecting,  processing,  analysis,  evaluation,  and  filing. 

•  Ensuring  that  DLE  team  members  comply  with  all  test  procedures. 


lA 


•  Ensuring  that  data  collected  are  adequate  and  thorough. 


•  Attending  conferences,  meetings,  and  demonstrations  as 
required. 

0  Presenting  logistics  briefings  as  required. 


ORGANIZATION  OF  THIS  CHAPTER 

ATD  hardware  operational  suitability  evaluations  encompass  a  wide 
variety  of  concerns.  Figure  2-1  shows  the  five  major  hardware  suita¬ 
bility  areas  of  concern  and  the  subelements  associated  with  each.  The 
remainder  of  this  chapter  is  organized  into  five  major  subsections  in 
accord  with  this  overall  structure.  In  order  of  presentation,  the 
evaluation  concerns  treated  are: 

A.  Reliability; 

B.  Maintainability; 

C.  Availability; 

D.  Logistics  Supportabil  ity;  and 

E.  Operating  and  Support  Costs. 

To  the  extent  practicable,  the  discussion  for  each  major  suit¬ 
ability  element  first  defines  that  element  in  the  context  of  ATD  OTAE. 
Next,  the  discussion  describes  the  evaluation  methods  to  be  used. 
Each  subsection  discussion  then  concludes  by  identifying  practical 
guidelines  that  may  serve  to  inform  the  test  director  of  possible  lead 
time  requirements,  phase  of  test  specific  concerns  (i.e.,  lOT&E/QOTAE, 
FOT&E),  or  general  aids  to  the  successful  accomplishment  of  the 
various  operational  suitability  assessments. 
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HARDWARE  OPERATIONAL 
SUITABILITY  EVALUATION 


Figure  2-L  Elements  of  hardv^are  operational  suitability  evaluation. 


A.  RELIABILITY 


RELIABILITY  EVALUATION  ELEMENTS 

Reliability  is  defined  as  the  probability  that  a  system  will  per¬ 
form  satisfactorily  for  a  given  period  of  time  when  used  under  stated 
conditions.  Reliability  thus  relates  to  the  frequency  with  which 
failures  occur  and  the  relationship  of  those  failures  to  the  mission 
of  the  system  in  question.  In  the  case  of  ATDs,  the  mission  is  that 
of  supporting  aircrew  training.  ATD  reliability,  therefore,  is  an 
index  of  how  satisfactorily  the  ATD  and/or  its  major  subsystems  will 
hold  up  when  used  for  training. 

Three  categories  of  ATD  reliability  are  of  interest--hardware 
(logistics)  reliability,  operational  (mission)  reliability,  and  the 
reliability  of  built-in  test  equipment  (BITE).  These  categories  are 
defined  briefly  below. 

Hardware  Reliability 


Hardware  reliability  (or  logistics  reliability)  refers  to  the 
identification  of  failure  patterns  or  trends  that  may  adversely  impact 
the  system's  functioning  and  that  will  require  excessive  resources  to 
restore  the  system  to  the  required  operating  condition.  Since  the 
intent  of  the  hardware  reliability  evaluation  is  to  determine  (or 
estimate)  the  downstream  logistics  impact  of  any  maintenance 
occurrence  in  terms  of  spares  requirements,  maintenance  manpower,  or 
support  test  equipment,  all  maintenance  occurrences  are  of  interest. 

Any  maintenance  actions  on  an  ATD,  including  preventive  main¬ 
tenance,  adjustments  (e.g.,  CRT  focus),  light  bulb  burn-out,  computer 
"glitch,"  etc.,  are  referred  to  as  occurrences.  Occurrences  that  have 
to  do  with  any  system/component  breakdown  or  deterioration  such  that 
normal  system  function  is  degraded  are  referred  to  as  failures.  Fail¬ 
ures  can  be  further  classified  as  either  critical  or  noncritical.  A 
critical  failure  is  a  failure  that  causes  a  significant  disruption  to 
the  ongoing  training  mission  of  the  ATD.  A  noncritical  failure  is  a 
failure  of  the  system  in  an  area  or  manner  that  does  not  affect  the 
ongoing  training  mission.  Thus,  hardware  reliability  addresses  the 
potential  effects  of  all  occurrences  which  can  place  a  demand  on  the 
logistics  support  system,  whether  or  not  training  mission  capability 
is  affected  by  the  occurrence. 

Operational  Reliability 


Operational  reliability  (or  mission  reliability)  is  a  measure  of 
the  ability  of  an  ATD  to  complete  its  planned  training  functions.  As 


a  consequence,  only  those  "critical  failures"  (as  defined  above)  that 
interrupt  or  degrade  significantly  the  capability  of  the  device  to 
support  specific  planned  or  ongoing  training  activities  are  of 
interest  in  the  determination  of  mission  reliability. 

Reliability  of  Built-in  Test  Equipment  (BITE) 


The  reliability  of  BITE  systems  refers  to  the  ability  of  those 
systems  to  detect  and  isolate  failures.  Examples  of  BITE  include, 
among  others,  computer  controlled  preflight  programs,  indicator  panels 
(open  circuit  breaker,  system  off,  etc.),  and  computer  system 
diagnostic  programs. 


RELIABILITY  EVALUATION  METHODS 

There  are  two  types  of  structured  reliability  evaluations.^  The 
first  type,  known  as  the  Fixed  Length  Test,  requires  that  the  system 
in  question  be  run  for  a  specified  number  of  hours.  If  more  than  a 
predetermined  number  of  failures  occur  during  that  period,  the  system 
fails  the  test;  otherwise,  it  passes.  The  second  type  of  test  is 
known  as  the  Probability  Ratio  Sequence  Test  (PRST).  In  this  test, 
the  system  is  operated  for  a  specified  minimum  time.  Beyond  this 
point,  determination  is  made  as  to  which  one  of  the  three  following 
conditions  exists;  (1)  If  an  upper  limit  of  failures  has  been 
exceeded,  the  system  has  failed  the  test;  (2)  if  the  number  of  fail¬ 
ures  is  below  some  lower  limit,  the  system  has  passed  the  test;  or  (3) 
if  the  number  of  failures  is  between  the  upper  and  lov;er  limits,  the 
test  must  continue.  It  is  this  second  type  of  reliability  test  (PRST) 
that  applies  most  often  to  ATD  T&E  in-plant. 

For  on-site  OT&Es,  a  structured  test  program  such  as  described 
above  does  not  typically  apply.  The  on-site  operational  environment 
itself  provides  a  situation  in  which  system  failure  data  are  recorded 
as  they  occur  on  a  daily  basis  for  the  entire  period  of  operational 
testing.  In  this  way,  a  more  credible  prediction  of  mature  system 
reliability  may  be  achieved. 

Data  Collection  Procedures 


The  data  collection  procedures  for  hardware  and  operational 
reliability  evaluations  are  basically  the  same.  Information  about  the 
occurrences  used  to  compute  reliability  indices  comes  from  the 
following  sources: 


^MIL-STD-781 ,  "Reliability  Design  Qualification  and  Production 
Acceptance  Tests." 
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Reliability  Demonstration  Plan.  This  is  the  basic  guideline 
for  conducting  the  reliability  test  during  QOT&E/IOT&E. 


•  Reliability  Test  Log.  This  record  is  used  to  recover  data  to 
assist  in  providing  logistic  support  and  to  determine  if  the 
ATD  meets  reliability  requirements. 

•  Component  Removal  Data.  These  data  are  used  in  determining 
the  need  for  special  tools,  procedures  and  test  equipment, 
tech  data,  special  skills,  etc.  Such  data  include; 

—  Operating  times 

—  Occurrence  data 

--  Repair  data 

—  Nonfailure  removal  data  (includes  instances  in  which 
there  is  indication  that  a  component  is  defective  or 
that  a  component  has  been  replaced,  but  the  problem 
still  exists) 

0  Contractor  Failure  Data 

0  Correlated  AFLC  D056  Reports  (these  are  historical  data  on 
equipment  in  the  inventory) 

0  Debriefing  Records 

0  USAF  Service  Reports  (SRs) 

0  Maintenance  Observation  Forms  (e.g.,  AFTEC  Form  99,  AFTO  349, 
or  similar) 

0  Reliability  and  Maintainability  Allocations,  Assessments,  and 
Analysis  Report  (DI-R-3535) 

0  Simulator  Acceptance  Deviation  or  Waiver  Reports  (or  like 
reports) 

0  Logistics  Support  Analysis  Record  (LSAR) 

Reliability  Test  Log.  The  principal  means  of  collecting  relia¬ 
bility  data  is  the  Reliability  Test  Log.  Collection  of  reliability 
data  involves  two  basic  elements  or  parts.  The  first  of  these  is  a 
chronological  account  of  events,  e.g.,  operating  on-  and  off-times, 
times  of  occurrences  or  failures,  and  maintenance  times.  An  example 


of  a  Reliability  Test  Log  is  depicted  in  Figure  2-2.  (The  features  of 
the  specific  ATD  under  test  will  dictate  the  precise  format  of  the  log 
headings.) 

A  separate  Reliability  Log  sheet  is  filled  out  for  each  day  of 
the  reliability  test.  An  event  number  is  assigned  to  each  maintenance 
action.  That  event  number  is  based  on  the  date/event  sequence  and  is 
used  to  aid  in  relating  the  chronological  accounting  in  the  supporting 
maintenance  observation  reports  used  to  gather  more  detailed  main¬ 
tenance  action  information.  Some  logs  will  be  "full indicating  a 
high  number  of  occurrences  during  a  day  (Figure  2-3),  while  others 
will  be  nearly  "empty,"  indicating  a  relatively  low  number  of 
occurrences  (Figure  2-4). 

Maintenance  Observation  Reports.  The  second  data  element  sup¬ 
porting  reliability  determinations  is  derived  from  maintenance  obser¬ 
vation  reports.^  (Either  AFTEC  Form  99  or  AFTO  Form  349  can  be  u'-ed 
for  this  purpose.)  AFTEC  Form  99  is  shown  in  Figure  2-5. 

Information  contained  on  the  maintenance  observation  report  is 
used  as  the  principal  source  for  later  maintainability,  tech  data, 
spares,  personnel,  facility,  etc.,  determinations.  Information  shown 
also  includes  How-Mal  codes  (I AW  AFTO  43-1-06-2),  maintenance  type 
codes,  time  to  restore,  and  other  critical  data. 

Maintenance  observation  reports  are  normally  filled  out  by  DLE 
maintenance  personnel  for  each  maintenance  action  performed.  In  some 
cases,  such  as  tests  with  contractor  maintenance,  the  ATD  contractor 
can  be  required  to  fill  out  the  necessary  maintenance  record  forms. 

BITE  reliability  data  collection.  To  calculate  the  required  BITE 
rates,  it  is  necessary  that  appropriate  notations  are  made  on  the 
reliability  log  and  maintenance  observation  reports.  In  some  cases,  a 
separate  BITE  log  may  be  more  applicable  (see  Figure  2-6).  Those 
notations  consist  of  coded  entries  to  the  log,  immediately  following 
the  description  of  the  occurrence.  Refer  to  Appendix  A  for  a  listing 
and  definition  of  the  BITE  rate  codes. 

Categorization  of  data.  Occurrences  during  any  in-plant  relia¬ 
bility  evaluation  must  be  classified  as  to  whether  they  are  countable 
or  noncountable  occurrences.  This  is  necessary  to  ensure  that  only 
countable  occurrences  are  included  in  the  calculations  of  reliability 
indices.  This  categorization  may  occur  at  the  time  of  the  occurrence 
or  later,  at  the  discretion  of  the  DLE.  The  two  categories  are  as 
follows : 


There  forms  are  also  commonly  known  as  Support  Evaluation 
Worksheets. 
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ARPTT  RELIABILITY  TEST  LOG 


DATE:  07/11/79  (311) 
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Figure  2-3.  Example  reliahility  test  log  showing 
a  number  of  occurrences. 
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RELIABILITY  TEST  LOG 


Power  up  and  daily  checks 

Start  preflight 

Finish  preflight 

Adjust  visual  and  finish  checks 

Start  Mission  #1 

Mission  #1  ends 

Changed  from  "H"  to  "G" 

Mi ssion  #2  No-show 
Mission  starts(#3) 

Mission  #3  ends 
Start  Mission  #4 
End  Mission  #4 
Start  verifications 
Stop  verifications 


Figure  2-4.  Example  reliability  test  log  showing 
no  occurrences. 


Figure  2-5.  Example  maintenance  observation  report.  (AFTEC 
Form  99  is  shown.  AFTO  Form  349  can  also  be 
used. ) 
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DIAGNOSTIC  TEST  LOG 


•  Countable  Occurrences.  Those  occurrences  on  operationally 
configured  equipment  which  the  test  team  determines  to  be 
representati ve  of  occurrences  that  might  happen  in  the  opera¬ 
tional  environment. 

•  Noncountable  Occurrences.  Those  occurrences  which  are  unique 
to  the  test  environment  and  which  are  not  representati  ve  of 
the  operational  environment.  For  example,  failures  which 
might  be  attributable  to  deficiencies  of  an  in-plant  test 
environment,  such  as  inadequate  air  conditioning,  would  be 
categorized  as  noncountable. 

Re liabil ity  Indices 

Mean  time  between  maintenance  action  (MTBM)  and  mean  time  be¬ 
tween  critical  failures  (MTBCF)  are  the  two  principal  measures  of 
reliability  determined  during  reliability  evaluations.  MTSM  indices 
have  to  do  with  the  hardware  reliability  of  the  device,  and  MTBCF 
indices  relate  to  the  operational  reliability  of  the  device.  The 
hardware  and  operational  reliability  determinations  are  calculated  by 
the  OLE  rel  iabil ity/maintainabil ity  engineer. 

Hardware  reliability.  There  are  six  versions  of  the  MTBM 

described  in  following  paragraphs,  all  of  which  are  calculated  by 
dividing  operating  time  by  the  total  number  of  ofi-equipment  main¬ 
tenance  occurrences,  where  operating  time  is  defined  as  the  system 
elapsed  time  indicated  (ETI).  The  formula  used  to  calculate  MTBM  is 
shown  in  Appendix  A.  The  six  versions  of  MTBM  are  as  follows: 

1.  MTBM  (preventive):  This  form  of  MTBM  considers  preventive 
on-equipment  occurrences  during  the  active  test.  (Adjustments, 
checking  fluid  levels,  "tweaking"  drifting  indicators,  etc.) 

2.  MTBM  (induced):  This  form  of  MTBM  considers  all  on-equipment 
occurrences  resulting  from  an  item  no  longer  meeting  performance 
requirements  because  of  an  induced  condition.  An  induced  condition  is 
defined  as  a  condition  resulting  from  an  external  source,  e.g.,  power 
fluctuation,  environmental  conditions,  or  operational  error.  Examples 
of  operational  errors  would  be  the  execution  of  an  improper  on- 
equipment  preventive  maintenance  action. 

3.  MTBM  (no  defect):  This  form  of  MTBM  considers  on-equipment 
occurrences  resulting  from  a  non-defect  condition.  These  are  "false 
alarm"  occurrences.  An  example  of  a  false  alarm  would  be  an  aiparent 
failure  such  as  might  be  indicated  by  operator  error/confusion  or  in  a 
number  of  component  areas  when  a  failure  of  a  power  supply  occurs. 


4.  MTBM  (inherent):  This  form  of  MTBM  considers  al)  on- 

equipment  occurrences  resulting  from  an  item  no  longer  mooting  perfor¬ 
mance  requi rements  due  to  an  inherent  condition.  An  inherent 
condition  is  defined  as  a  condition  resulting  from  an  internal  degra¬ 
dation  of  a  component.  A  common  example  would  be  a  degradation  in  IC 
"chip"  function  due  to  a  defective  batch  or  due  to  heat.  Once  the 

defective  chips  were  replaced,  the  system  would  function  normally. 

MTBM  (inherent)  on-equipment  occurrences  are  computed  using  all  How- 

Mal  codes  excluding  those  for  MTBM  (induced)  or  MTBM  (no  defect). 

5.  MTBM  (all  failures):  This  form  of  MTBM  considers  all  on- 
equipment  failures  and  refers  to  all  failures  considered  in  MTBM 
(inherent),  MTBM  (induced),  and  MTBM  (no  defect). 

6.  MTBM  (total  corrective  maintenance):  This  form  of  MTBM  con¬ 
siders  on-equipment  occurrences  of  MTBM  (inherent)  and  MTBM  (induced). 

Operational  reliability.  The  F'iean  Time  Between  Critical  Failures 
(MTBCF)  is  an  index  of  the  operational  mission  reliability  of  the  ATD. 
MTBCF  is  the  total  operating  time  during  the  evaluation  divided  by  the 
total  number  of  critical  failures  during  that  time. 

As  defined  earlier,  a  critical  failure  is  one  that  degrades  the 
specified  ongoing  training  mission  of  the  device.  When  a  failure 
occurs,  the  instructor/operator  pilot  at  the  trainee  station  is  con¬ 
sulted  to  determine  whether  the  failure  had  a  significant  impact  on 
the  training  mission  occurring  at  that  point  in  time.  An  example  of  a 
critical  failure  would  be  a  failure  in  the  Horizontal  Situation 
Indicator  during  instrument  training.  Such  a  failure  would  impact  the 
ongoing  training  mission,  and,  thus,  would  be  categorized  as  a 
"critical  failure."  A  failure  in  the  ATD  visual  system,  however,  pro¬ 
bably  would  not  have  affected  the  on_^oing  instrument  training  and, 
therefore,  would  not  be  classified  as  a  "critical  failure."  See 
Appendix  A  for  more  discussion  on  MTBCF  calculation  and  critical 
failure  distinctions. 

Built-in  Test  Equipment  (BITE)  reliability.  Quantitative  indices 
of  the  ability  of  BITE  to  detect  and  isolate  failures  accurately  are 
expressed  in  terms  of  various  BITE  rates.  The  following  BITE  rates 
are  calculated.  Computational  formulas  for  these  BITE  rates  are  con¬ 
tained  in  Appendix  A; 

•  BAIP  rate  (BITE  accurately  isolated  problem)  is  the  effec¬ 
tiveness  of  the  system  both  to  detect  a  failure  and  to  deter¬ 
mine  what  failed. 

•  BADP  rate  (BITE  accurately  detected  problem)  is  the  effec¬ 
tiveness  of  the  system  to  identify  that  a  particular  failure 
has  occurred. 


•  BFDP  rate  (RITE  failed  to  detect  problem)  is  the  occurrence 

rate  of  failures  that  should  have  been  determined  by  BITE  but 
were  not. 

•  BFA  rate  (BITE  false  alarm)  is  the  occurrence  rate  of  BITE 
identifying  failures  erroneously. 

Two  of  the  identified  reliability  indices  are  of  particular 

interest  to  the  ATD  OT&E  test  director:  (1)  the  overall  hardware 
reliability  index,  MTBM-all  failures  and  (2)  the  operational  relia¬ 
bility  index,  MTBCF.  After  these  two  values  have  been  calculated, 
they  are  compared  with  the  reliability  criteria  levels  as  defined  in 
the  Reliability  Test  Plan:  e.g..  Threshold,  Standard,  and  Goal.  Two 
actions  are  normally  undertaken  following  instances  where  the  data 
yield  a  reliability  value  less  than  threshold  value:  (1)  further 
reliability  analysis;  and/or  (2)  submission  of  a  Service  Report  (SR). 
The  purpose  of  the  follow-on  reliability  analysis  is  to  identify  the 
subsystem,  equipment,  or  component  which  is  the  principal  contributor 
to  or  direct  cause  of  the  reliability  problem.  The  Service  Report  is 

the  official  means  of  notifying  the  SimSPO  of  a  deficiency  and  has  as 

its  purpose  the  initiation  of  corrective  action  to  resolve  the  prob¬ 
lem.  SRs  remain  "active"  until  cleared  by  whatever  corrective  action 
is  determined  to  be  appropriate.  The  remaining  five  reliability 
indices--preventive,  inherent,  induced,  no  defect,  and  total 
corrective--are  principally  of  use  in  validating  and  refining  pre¬ 
viously  developed  estimates  (see  Figure  2-7,  Example  reliability 
reporting  table). 


RELIABILITY  MEASURE  VALUE  (HOURS)  THRESHOLD  STANDARD  GOAL 

MTBCF  45.2  17.2  39.0  49.0 

MTBM  (Preventive)  15.0 

MTBM  (Induced)  115,8 

MTBM  ( Inherent)  25.3 

MTBM  (Total  Corrective)  19.3  12.3  21.5  43.0 

MTBM  (No  Defect)  50.7 

MTBM  (All  Failures)  12.5 


Figure  2-7.  Example  reliability  reporting  table  (example  shows 
hypothetical  data).  Note:  Formats  for  reporting 

should  follow  current  AFR  80-5  "Reliability  and 
Maintainability  Programs  for  Systems,  Subsystems, 
Equipment,  and  Munitions." 


Supplementary  reliability  data  for  equipment  items  which  exhibit 
high  failure  rates  during  the  evaluation  period  and  which  are  pre¬ 
sently  in  the  USAF  inventory  can  be  obtained  from  the  AFLC  D056  pro¬ 
duct  performance  system  data  bank  and  AFLC  G026  Material  Improvement 
Program  (MIP)  report(s).  Correlation  of  the  OT&E  data  with  those 
historical  data  can  aid  the  DLE  test  team  in  making  more  accurate 
estimates  of  system  and  subsystem  reliability. 


RELIABILITY:  TEST  DIRECTOR  CONCERNS 
Phase  of  Test  Considerations 

Test  and  field  data  covering  a  variety  of  systems  have  indicated 
that,  if  the  system  is  mature,  the  failure  rate  will  be  relatively 
constant  throughout  its  programmed  operational  life  cycle.  When 
equipment  is  produced  and  first  introduced  into  the  inventory,  there 
are  usually  more  failures  during  a  debugging  or  burn-in  period.  Like¬ 
wise,  when  the  equipment  reaches  a  certain  age,  there  is  a  wearout 
period  during  which  failure  rates  increase.  A  typical  failure-rate 
curve  illustrating  this  point  is  depicted  in  Figure  2-8.  In  accord 
with  this  relationship,  reliability  assessments  early  in  a  systems 
operational  life  are  likely  to  result  in  lower  MTBM  and  MTBCF  indices 
than  would  actually  be  the  case  when  the  system  matures.  In-plant 
test  data,  for  example,  will  likely  show  lov/er  system  reliability  than 
data  collected  during  subsequent  on-site  tests.  One  approach  for 
dealing  with  the  problem  of  estimating  mature  system  values  is  to  com¬ 
pile  separate  data  for  the  latter  portion  of  the  overall  test  period. 

MATURE  SYSTEM 
CONSTANT  FAILURE 
RATE  REGION 

DECREASING  FAILURE  INCREASING  FAILURE 

RATE  DURING  BURN-  r~L,  DURING  WEAR- 

IN  PERIOD  PERIOD 


o  o 


In  some  cases,  improvement  in  mature  system  values  mav  be  attri¬ 
butable  to  fewer  failures  and  malfunctions  in  a  single  critical  sub- 
system,  e.g.,  computer  system,  visual  system.  ATDs  frequently 
incorporate  subsystems  from  various  manufacturers.  If  the  reliability 
of  one  of  those  critical  subsystems  is  low  during  early  test  phases 
because  of  faul ty  components ,  defective  cabl ing ,  etc . ,  it  is  likely 
that  attention  by  the  appropriate  contractor' s  technical  represen¬ 
tative  will  alleviate  the  problem  for  later  test  phases. 

personnel  Roqui rements 

A  number  of  factors  have  an  impact  on  the  numbers  and  types  of 
personnel  that  will  be  required  for  reliability  determinations  during 
AID  OTSE.  Among  others,  these  include  device  complexity  and  con¬ 
figuration,  phase  of  test,  number  of  test  items,  and  site  of  test.  In 
general,  the  following  guidelines  apply;  however,  these  personnel 
guidelines  must  be  modified  by  the  test  director  as  appropriate  for 


his  particular  circumstances; 

SPECIALTY 

SOURCE 

AFSC 

NUMBER 

DLE 

AFLC  or 
MAJCOM 

66170 

34100 

1 

Rel  iabil ' ty  S 
Maintainabil ity 
(R&M)  Engineer 

AFl.c: 

2895 

1 

FI ight  Sim  Tech 

MAJCOM 

34174 

1  /station/ Shi  ft 

Offensive  Sim  Tech 

MAJCOM 

34176 

1 /station,' shi  ft 

Defensive  Sim  Tech 

MAJCOM 

34172 

1 /station/ shi ft 

For  in-plant  tests  (e.g.,  combined  0T<§E/I0r(&E ) ,  the  reliability 
demonstration  (conducted  by  ASO)  normally  occurs  at  the  end  cf  the 
total  acceptance/qualification  test  period.  An  AFLC  reliability 
engineer  is  not  needed  for  the  total  test  period,  but  only  for  the 
reliability  demonstration .  A  test  may  be  scheduled  to  last  about  a 
month,  but  because  of  slippages  in  the  program  and  ongoing  engineerinq 
development  on  the  device,  it  may  run  th»^oe  or  four  times  that  long. 
Proper  resource  identification  and  TDY  schedul ing/budgeti ng  are  par¬ 
ticularly  important  in  this  case,  because  a  lengthy  slip  in  the 
reliability  demonstration  can  create  a  drain  on  limited  IDY  funding 
resources . 

The  reliability  demonstration  is  not  always  conductnd  in-p1ant 
In  some  instances,  the  Air  Force  may  conduct  it  after  the  dcwice  has 
arrived  on-site.  In  this  case,  the  reliability  demonstration  v;ou1d 
actually  be  accomplished  on-site  as  the  last  event  prior  to  acceptance 
( DD  2^)0  sign-off)  , 
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A  Logistics  Command  reliability  engineer  is  typically  required 
only  for  the  structured  reliability  denonstration .  A  reliability 
engineer  is  not  normally  required  for  extended  on-site  lOTAE  or  FOTAE 
period  of  testing.  However,  it  should  be  noted  that,  prior  to  the 
start  of  test,  a  reliability  engineer  should  be  consulted  to  assure 
that  adequate  data  collection  procedures  have  been  defined. 


B.  MAINTAINABILITY 


MAINTAINABILITY  EVALUATION  ELEMENTS 

There  are  two  basic  kinds  of  maintenance  activity.  The  first, 
corrective  maintenance,  occurs  whenever  a  system  component  or  function 
has  failed  and  requires  subseguent  repair  or  restoration.  The  other 
type,  preventive  maintenance  (or  scheduled  maintenance),  occurs  on  a 
regular  basis  and  is  intended  as  a  means  of  reducing  or  preventing 
unplanned  downtime  for  corrective  maintenance. 

Maintainability  is  concerned  with  those  characteristics  of  system 
design  and  installation  which  affect  the  ease  or  difficulty  of  these 
maintenance  actions — corrective  and  preventive--so  that  the  system  may 
be  restored  to,  or  retained  in,  a  specified  working  condition. 
Requirements  for  maintainability  considerations  are  formalized  in 
Military  Standards  470  and  471  and  are  imposed  on  all  contractors 
supplying  equipment  to  the  mil  itary 

ATD  maintainability  and  reliability  are  closely  interrelated  in 
determining  the  overall  suitability  of  a  system.  It  is  possible,  for 
example,  to  have  a  system  that  meets  its  reliability  requirement  in 
terms  of  number  of  failures,  but  does  not  meet  the  maintainability 
requirement  in  terms  of  the  time  required  to  restore  the  system  to 
operational  status.  ThereforFj  inclusion  of  maintainability  eval¬ 
uations  during  ATD  OT&Es  is  an  essential  part  of  the  evaluation  of  ATD 
operational  suitability. 

Quanti tati ve  Maintai nabil  i  ty 

There  are  two  categories  of  maintainability  of  interest  during 
ATD  OTAE.  The  first  of  these,  quantitative  maintainability,  is 
defined  as  a  measure  of  the  maintenance  time  and  resources  required  to 
keep  an  item  operating.  Quantitative  maintainabil ity  evaluations 
include  computations  of  mean  time  to  repair  (MTTR),  maintenance 
manhours  per  operating  hour  (MMH/OH),  mean  manhours  to  repair  (MMTR), 
maintenance  manhours  per  mission  (MMH/M),  maintenance  manhours  per 
training  hour  (MMH/TH),  and  maintenance  hours  per  action  taken  code 
(i.e.,  the  portion  of  total  corrective  maintenance  time  spent  on  each 
action  taken  category:  troubleshooting,  repair,  adjustment,  remove 
and  replace,  and  no  defect).  These  indices  provide  the  test  director 


^MIL-STD-470,  "Maintainability  Program  Requirements." 

2 

MIL-STD-471 ,  "Maintainabil ity  Veri fication/Demonstration/ 
Eval uation." 


with  a  numerical  basis  for  drawing  conclusions  about  the  overall  main¬ 
tainability  of  the  AID  in  question  and  serve  to  highlight  those  areas 
that  may  require  further  study. 

Qualitative  Maintainability 


The  second  category  of  maintainability,  qualitative  main¬ 
tainability,  refers  to  those  areas  of  system  maintainability  that  must 
be  evaluated  by  qualitative  judgments.  For  example,  item  location 
accessibility,  safety  hazards,  and  "serviceability"  would  be  of  con¬ 
cern  in  qualitative  maintainability  evaluations.  Human  factors 
concerns--e.q . ,  weight,  handles,  height  above  ground  level--would  also 
be  pertinent  in  the  qualitative  evaluation.  Guidelines  for  maintain¬ 
ability  human  factors  considerations  are  contained  in  MIL-STD-1472, 
"Human  Engineering  Design  Criteria  for  Military  Systems,  Equipment, 
and  Facilities,"  Section  5.9  of  which  covers  the  following  main¬ 
tainability  design  guidelines: 

•  Mounting  of  components  within  units 

•  Adjustment  control s 

•  Accessibility 

•  Lubrication 

•  Unit  cases  and  covers 

•  Access  openings  and  covers 

•  Fasteners 

•  Unit  design  for  efficient  handling 

•  Mounting 

•  Retaining  devices 

•  Conductors 

•  Connectors 

•  Test  points 

•  Test  equipment 

•  Failure  indications  and  fuse  requirements 

•  Gas  and  fluid  line  identification 
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MAINTAINABILITV  EVALUATION  METHODS 


The  OLE  should  review  all  maintainability  test  plans  to  ensure 
that  the  planned  testing  and  data  recovery  procedures  are  sufficient 
to  allow  for  an  adequate  assessment  of  quantitative  and  qualitative 
corrective  and  preventive  maintainability  concerns.  The  DLE  team 
should  actually  observe  contractor  maintenance  tasks  during  main¬ 
tainability  evaluations  to  ensure  that  available  technical  data  and 
proper  support  equipment  are  being  used  and  that  all  repair  data  are 
being  thoroughly  and  accurately  recorded. 

Quantitative  Maintainability  Evaluation  Procedures 

Depending  on  the  user's  concept  of  maintenance--Air  Force  or 
contractor--early  OT&E  (lOT&E  or  QOT&E)  maintainability  determinations 
may  be  conducted  either  in-plant  or  on-site.  If  organic  Air  Force 
maintenance  is  planned,  the  maintainability  evaluation  will  normally 
be  conducted  on-site  in  its  intended  environment  with  representative 
Air  Force  maintenance  personnel.  If  contractor  logistic  support  (CLS) 
is  planned,  the  maintainability  evaluation  will  normally  be  conducted 
in  the  contractor’s  facility.  Two  sources  of  maintainabil ity  data  may 
be  combined  to  support  early  OT^iE  estimates  in  this  area.  The  first 
of  these  is  the  stri/ctured  maintainability  doiionstration  (M-demo)  con¬ 
ducted  by  the  SimSPO  during  device  DT4E  or  QT&E.  The  second  source  is 
those  maintainability  data  resulting  from  maintenance  actions  per¬ 
formed  during  other  periods  of  test  (e.g.,  reliability  testing,  opera¬ 
tional  effectiveness  testing,  etc.). 

M-demo  procedures.  The  following  is  a  description  of  the  basic 
procedures  followed  in  the  structured  ASD  maintainability  demonstra¬ 
tion.  However,  collection  of  data  during  other  periods  usually 
follows  the  same  basic  methods  of  observing  maintenance  actions  and 
recording  relevant  information  on  a  standardized  maintenance  obser¬ 
vation  record  form  (e.g.,  AFTO  349).  The  principal  difference  is  in 
the  source  of  "faults."  Instead  of  inserting  preselected  faults  for 
evaluation  as  is  done  in  M-demo,  observations  during  other  occasions 
are  based  upon  those  "natural"  faul ts  and  malfunctions  that  may  occur. 

In  gathering  data  to  assess  the  device's  maintainability  during 
M-d0no  in  the  early  phases  of  OT&E,  one  or  more  preselected  faults  are 
inserted  into  the  equipment  with  the  maintenance  team  absent  from  the 
area.  The  faults  inserted  are  identified  in  a  set  of  "failure  data 
forms"  supplied  by  the  contractor,  and  approved  by  the  SimSPO.  Figure 
2-9  illustrates  the  format  and  content  of  these  forms.  Once  a  fault 
has  been  inserted  and  verified,  the  maintenance  team  enters  the  area 
and  initiates  the  appropriate  corrective  maintenance  action.  An  inde¬ 
pendent  observer  uses  a  stop  watch  and  a  test  observer  recording  form 
(AFTEC  99,  AFTO  349)  to  record  the  time  required  for  each  step  in  the 
maintenance  process. 


CORRECTIVE  MAINTENANCE  TASK  TO  BE  DEMONSTRATED 


FAILURE  NO.  1 

NOMENCLATURE: 

Matrix  board  (VASI  drive) 

A  74  /4  89/6  00-6  2  0 

LOCATION  OF  FAILURE: 

Maintenance  rack 

FAILURE  INSERTED: 

Remove  board 

Lift  off  link  between  T35  and  AA33 
TYPE  OF  FAILURE  INSERTED: 

Failure  of  matrix  board  due  to  open  circuit  condition 
SYMPTOMS; 

No  VASI  motor  drive  effective  on  VASI  l.b.,  {parallel) 
Prism  may  exhibit  hue  neither  red  nor  white 

ESTIMATED  REPAIR  TIME:  43  minutes 


Figure  2-9.  Example  contractor  provided  maintainability  failure  data 
form. 


Separate  data  sheets  are  filled  out  for  each  corrective  task  or 
simulated  failure.  The  failure  data  forms  and  the  completed  test 
observer  record  forms  constitute  the  bulk  of  the  maintainability 
demonstration  raw  data.  It  should  be  noted  that  some  portions  of  the 
data  collected  on  these  forms  are  quantitative,  (e.g.,  time),  whereas 
other  portions  are  qualitative  (e.g.,  use  of  tools,  test  equipment, 
etc.) . 

In  some  cases,  failure  to  meet  time  standards  for  restoring  the 
system  to  operation  can  be  traced  to  support  factors  such  as  an 
excessive  use  or  lack  of  technical  documentation,  test  equipment,  or 
other  unproductive  time  use.  These  factors  should  be  addressed  under 
logistics  supportabil i ty  assessments. 

The  above  described  data  gathering  procedure  applies  to  both 
corrective  and  preventive  M-demos.  Time  data  for  preventive  main¬ 
tenance  tasks  being  evaluated  are  recorded  in  the  same  manner  and  on 
the  same  forms  as  are  used  for  "simulated"  failures,  except  that  the 
maintenance  team  is  instructed  to  initiate  a  specified  preventive 
maintenance  routine  as  opposed  to  a  corrective  maintenance  action. 

Maintainability  testing  typically  lasts  a  minimum  of  two  weeks, 
during  which  time  a  number  of  corrective  and  preventive  maintenance 
tasks  are  demonstrated.  Usually  the  majority  of  the  tasks  to  be 
demonstrated  were  randomly  selected  in  advance  by  the  SimSPO  from  a 
larger  set  of  tasks.  However,  in  addition  to  the  randomly  preselected 
tasks,  the  Air  Force  often  may  select  a  number  of  additiona''  main¬ 
tenance  tasks  for  demonstration  purposes. 

As  noted  earlier,  the  OT&E  maintainability  evaluation  includes 
both  data  resulting  from  the  structured  M-deno  and  data  resulting  from 
natural  maintenance  actions.  Later  FOT&E  focuses  exclusively  upon 
maintenance  actions  resulting  from  natural  faults  and  malfunctions  as 
they  occur. 

Quantitative  Maintainability  Indices 

Mean  time  to  repair  (MTTR)  and  maintenance  manhours  per  operating 
hour  (MMH/OH)  are  the  two  primary  measures  of  effectiveness  which 
result  from  the  quantitative  reliability  evaluation.  Other  measures 
are  mean  manhours  to  repair  (MMTR)  and  maintenance  manhours  per 
training  hour  (MMH/TH),  Computational  formulas  for  these  measures  are 
contained  in  Appendix  B. 

Mean  time  to  repair  (MTTR).  MTTR  is  the  total  corrective  main¬ 
tenance  clockhours  during  the  test  period  divided  by  the  total  correc¬ 
tive  maintenance  actions  over  that  same  period.  (Certain  guidelines 
have  been  developed  regarding  which  times  to  include  in  MTTR  calcula¬ 
tions.  These  guidelines  are  contained  in  Appendix  B.)  Time  is 
usually  reported  to  the  nearest  tenth  (0.1)  hour. 


MTTR  ( cri tical ) .  This  is  a  measure  of  the  corrective  manhours 
expended  on  critical  failures  divided  by  the  number  of  critical 
failures.  This  provides  the  average  time  to  repair  a  critical 
failure.  Maintenance  crew  size  is  not  considered.  Separate  MTTR 
(critical)  values  may  be  compiled  for  certain  device  subsystems  in 
addition  to  the  device  as  a  whole.  For  example,  visual  system,  motion 
system,  computer  system,  and  instructor/operator  station  usually  are 
compiled  as  separate  entities. 

Mean  manhours  to  repair  (MMTR).  MMTR  is  the  total  corrective 
maintenance  manhours  during  the  test  period  divided  by  the  total 
corrective  maintenance  actions  over  that  same  period.  The  time  guide¬ 
lines  specified  for  MTTR  apply  (see  Appendix  B) . 

Maintenance  manhours  per  training  hour  (MMH/TH).  WH/TH  is  the 
total  of  all  maintenance  manhours  expended  during  operational  testing 
divided  by  total  number  of  hours  the  ATD  was  used  for  operational 
training  (simulated  or  actual)  during  that  testing  period.  In  addi¬ 
tion  to  the  time  guidelines  specified  for  ^f^TR,  some  additional  con¬ 
siderations  are  defined  (see  Appendix  B) . 

Maintenance  manhours  per  operating  hour  (MMH/OH).  MIH/OH  is  the 
total  direct  maintenance  manhours  divided  by  the  total  operating  time 
of  the  system.  Maintenance  manhours  are  the  total  of  the  direct  main¬ 
tenance  manhours  expended  during  operational  testing  times  (e.g., 
reliability  demonstration  and  operating  effectiveness  testing)  on 
those  occurrences  listed  under  rTTBM.  Operating  hours  are  the  total  of 
the  system  power-on  hours  during  the  operational  testing  times. 
Manhours  are  reported  for  both  on-  and  off-equipment  expenditures. 
MMH/OH  is  calculated  for  all  manhours  expended  in  the  six  basic  cate¬ 
gories  of  maintenance  occurrences  (Preventive,  Inherent,  Induced,  No 
Defect,  All  Failures,  and  Total  Corrective),  and  also  for  inspection 
and  support  general  expenditures. 

Once  calculated,  the  results  of  quantitative  maintainability 
measures  are  evaluated  with  respect  to  the  threshold,  standard,  and 
goal  criteria  established  in  the  test  plan  (assignment  of  evaluation 
criteria  normally  applies  only  to  the  MTTR  and  MTTR  [critical] 
measures).  Figure  2-10  shows  an  example  of  how  these  values  may  be 
C'Sported.  For  those  items  causing  below  threshold  values,  comparisons 
'should  be  made  to  corresponding  reliability  data,  and  a  prioritized 
list  of  candidates  for  corrective  action  should  be  developed.  At  this 
point,  consideration  is  given  to  the  practical ity/feasibil ity  of 
correcting  the  condition,  and  a  service  report  (SR)  may  be  submitted 
as  a  result  of  this  consideration. 

Dual i tative  Maintainability  Data  Collection  Procedures 


As  noted  earlier,  qualitative  maintainability  determinations  have 
to  do  with  accessibility,  serviceability,  ease  of  maintenance,  human 
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MAINTAIMABTim' 

MEASURE 

— rOTAL'" 
TEST 

“  MATimr ' 

SYSTEM 

BijUinn 

stan¬ 

dard 

GOAL 

MHR 

1.50 

1.01 

4.0 

1.5 

0.7 

MTTR  (critical) 

4.35 

1.98 

4.0 

1.5 

0.7 

MHR 

3.10 

1.58 

MMH/TH 

0.43 

0.34 

MMH/OH 

-  Preventive 

0.20 

0.24 

-  Induced 

0.03 

0.02 

-  No  Defect 

0.01 

0.02 

-  Inherent 

0.21 

0.10 

-  All  Failures 

0.19 

0.13 

-  Total  Corrective 

0.31 

0.15 

SUBSYSTEM  MTTR  (critical) 


Computing  system 

8.60 

0.88 

Visual  system 

3.96 

2.39 

Motion  system 

1.23 

1.10 

Instructor  stations 

3.15 

2.85 

Figure  2-10.  Maintainabil  ity  reporting  format  (hypothetical  data). 
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factors,  etc.  The  judgments  made  during  this  type  of  evaluation  are 
not  necessarily  supported  by  strict  quantitative  criteria;  they 
require  that  the  evaluator  utilize  his  expertise  and  experience. 
Emphasis  is  placed  on  identifying  equipment  design  and  installation 
characteristics  that  have  potential  for  causing  maintenance  dif¬ 
ficulties  or  safety  hazards.  Therefore,  each  deficiency  or  potential 
deficiency  must  be  assessed  on  a  case-by-case  basis.  In  cases  where 
problems  are  identified  with  accessibility  and  location,  it  may  be 
possible  to  substantiate  resulting  SRs  with  quantitative  data.  Often 
still/motion  pictures  or  video  taping  may  prove  useful. 

The  procedure  employed  to  gather  qualitative  maintainab’’,  ity 
information  is  implemented  by  the  Maintainability  Evaluation  Check¬ 
list.  An  example  of  such  a  checklist  is  provided  in  Appendix  B. 
Elements  of  that  checklist  can  vary  significantly  in  detail  or  gener¬ 
ality.  In  OT&E,  the  OLE  team  reliability/maintainability  engineer 
manually  compiles  and  analyzes  the  required  data. 

Built-in  test  equipment  (BITE)  effectiveness.  Bite  effectiveness 
refers  to  tTie  impact  of  BITE  oTt  the  ma i n ta i nab i  1  i ty  of  the  system. 
The  various  BITE  rates  compiled  under  BITE  reliability  assessments 
should  be  assessed  to  determine  areas  where  BITE  significantly 
enhances  or  degrades  ease  of  maintenance  actions.  This  is  primarily  a 
qualitative  assessment  based  upon  the  types  of  BITE  available  and 
those  maintenance  actions  to  which  they  apply. 


MAINTAINABILITY;  TEST  DIRECTOR  CONCERNS 
Phase  of  Test  Considerations 

As  noted  in  the  reliability  subsection,  test  and  field  data  have 
indicated  that  such  measures  on  "mature"  systems  are  normally  improved 
over  measures  taken  when  the  system  is  first  introduced  into  the 
inventory.  Improvement  in  mature  system  maintainability  values  can  be 
expected  also  to  occur.  As  familiarity  is  gained  with  the  system  and 
common  maintenance  actions  are  defined,  along  with  properly  developed 
maintenance  technical  data,  the  mean  time  required  to  repair  the 
system  should  decrease,  thereby  resulting  in  improved  overall  MTTR  and 
MMH/OH  measures. 

The  technique  of  compiling  separate  data  for  the  latter  portion 
of  the  overall  test  period  may  also  be  employed  to  estimate  mature 
system  maintainability.  Figure  2-9,  presented  earlier,  portrays  how 
these  data  can  be  displayed  for  cotnparison  purposes. 

Personnel  Requirements 

Because  maintainability  tests  normally  occur  subsequent  to 
reliability  tests  during  combined  testing,  the  personnel  requirements 


34 


to  accomplish  maintainabil ity  assessments  are  not  usually  of  concern. 
Additional  simulator  technicians  are  not  normally  needed  because  the 
same  people  required  to  support  reliability  assessments  may  be  uti¬ 
lized  for  maintainability  tests.  It  is  important  that  these  personnel 
be  as  familiar  as  possible  with  the  maintenance  tasks  under  test.  The 
following  personnel  guidelines  generally  apply: 


SPECIALTY 

SOURCE 

AFSC 

NUMBER 

DLE 

AFLC  or 

66170 

1 

MAJCOM 

34100 

Rel iabil ity  & 

AFLC 

2895 

1 

Maintainabil ity 
(R&M)  Engineer 

Flight  Sim  Tech 

MAJCOM 

34174 

1 /station/shi ft 

Offensive  Sim  Tech 

MAJCOM 

34176 

1/station/shi ft 

Defensive  Sim  Tech 

MAJCOM 

34172 

1/station/shi ft 

As  was  the  case  with  the  reliability  engineer  during  reliability 
testing,  the  maintainability  engineer  is  not  needed  during  the  total 
test  period.  He  is  needed  only  for  the  maintainability  demonstration 
during  combined  testing.  During  FOT&E,  a  senior  logistics  person  may 
be  substituted  and  given  responsibility  for  maintainability  deter¬ 
minations.  However,  during  FOT&E,  maintainability  observation  may 
necessitate  augmenting  manpower  during  such  observation  periods 
depending  on  device  complexity  and  extent  of  test  requirements. 


C.  AVAILABILITY 


AVAILABILITY  EVALUATION  ELEMENTS 

The  availability  of  an  ATD  is  a  function  of  its  combined  relia¬ 
bility,  maintainability,  and  logistics  supportabil ity.  From  the 
standpoint  of  the  user,  ATD  availability  measures  reflect  the  readi¬ 
ness  of  the  ATD  to  perform  its  specific  training  mission  at  any  given 
point  in  time.  Availability  is,  therefore,  the  measure  of  greatest 
interest  to  the  operational  users  of  ATDs. 

There  are  two  basic  kinds  of  availability  addressed  during  ATD 
OT&E:  Inherent  Availability  (Ai)  and  Scheduling  Availability  (As). 
In  addition  to  these  two  kinds  of  availability  determinations.  Mission 
Capable  Rates  (MCRs)  are  of  interest  during  ATD  suitability  eval¬ 
uations.  The  following  paragraphs  discuss  availability  assessment 
procedures  (A,-  and  A5)  and  MCR  determinations. 

Inherent  Availability 


Inherent  availability  (A,-)  is  defined  as  the  probability  that  the 
simulator  will  operate  satisfactorily  at  a  given  point  in  time. 
Inherent  availability  measures  provide  preliminary  information  on  the 
potential  availability  of  the  device.  The  term  "potential"  is  used 
here  because  inherent  availability  assumes  an  artificial  environment 
in  which  there  are  no  logistics  delays,  free  time,  administrative 
time,  or  storage  time.  Inherent  availability  is  a  measure  of  the 
built-in  availability  of  the  device. 

Scheduling  Availability 


Scheduling  availability  (A^)  may  be  defined  as  the  probability  of 
completing  any  scheduled  training  mission.  Two  forms  of  Ag  are  used; 
The  first,  A5I,  is  the  ratio  between  completed  missions  and  scheduled 
missions;  the  second,  A^?,  is  the  ratio  between  hours  flown  and  hours 
scheduled.  It  goes  beyond  inherent  availability  to  include  the  actual 
environment  with  its  scheduling  and  logistics  delays. 

Mission  Capable  Rates 


Although  not  an  availability  estimate  in  the  pure 
capable  rates  (MCR)  are  also  a  critically  important 
availability  concerns.  Mission  Capable  Rates  predict 
of  possessed  time  that  a  device  can  be  expected  to 
training,  i.e.,  mission  capable.  In  this  sense,  MCR 
can  provide  a  truer  picture  of  device  availability  in 
its  operational  training  environment. 


sense,  mission 
aspect  of  ATD 
the  percentage 
be  usable  for 
determinations 
the  context  of 


AVAILABILITY  EVALUATION  METHODS 
Inherent  Availability 


Reliability  (MTBCF)  and  maintainability  (MTTR)  data  are  used  to 
calculate  A-j  for  the  overall  system  and  its  major  subsystems.  The 
data  collection  procedures,  definitions,  criteria,  and  miscellaneous 
factors  described  under  the  preceding  reliability  and  maintainabi  1  i  ty 
sections  also  apply  to  these  calculations.  System  MTBCF  and  MTTR  of 
critical  failures  are  used.  If  system  A-j  is  less  than  threshold, 
A-j  is  calculated  for  critical  subsystems  to  determine  which  subsystem 
is  responsible  for  the  low  Aj .  The  components  causing  the  low  Aj  are 
then  identified  and,  using  the  Aj ,  MTBCF,  and  MTTR  figures  as  justifi¬ 
cation,  a  service  report  (SR)  may  be  submitted.  Typical  thresnold 
values  for  availability  are  displayed  in  Figure  2-11.^  Inherent 
availability  indices  are  reported  in  all  interim  and  final  UTiE 
reports.  Figure  2-12  shows  an  example  of  how  Aj  data  may  be  foniiatted 
in  final  reporting. 

Scheduling  Availability 

Scheduling  availability  determinations  will  be  based  solely  on 
data  from  those  times  when  the  simulator  is  undergoing  integrated 
mission  testing.  The  DLE  reports  scheduling  availability  usual''y  on  a 
weekly  basis.  A  preprinted  mission  schedule  is  required  by  the  logis¬ 
tics  technicians.  This  schedule  should  show  the  types  of  missions  and 
the  hours  scheduled  per  type  of  mission.  The  instructor  pilot  (IP) 
determines  whether  a  training  mission  was  successful,  or  what  portion 
of  the  mission  was  successful.  Only  those  training  missions  which 
occurred  during  full  system  operational  tests  are  counted  as  scheduled 
or  successful  missions.  All  scheduled  training  time  lost  is  counted, 
including  losses  due  to  maintenance,  operations  "cancels”  or  "no 
shows,"  software,  and  facilities.  A^l  and  AcB  value  are  calculated, 
based  on  the  data  provided  on  mission  scheduling,  using  the  fonnulas 
in  Appendix  C. 

Scheduling  availability  may  be  calculated  for  each  type  of  mission 
flown  in  the  simulator.  Reasons  for  less  than  threshold  As  are  ana¬ 
lyzed.  Causes  of  the  low  As  are  sought  from  examination  of  the  relia¬ 
bility  and  ma i nta i nabi 1 i ty  data.  Causes  of  the  low  As  are  identified, 
and  a  new  As  calculated  whenever  the  cause  of  a  given  low  As  has  been 
corrected.  Service  reports  are  then  submitted  as  required. 

The  ULE  reports  overall  simulator  and  mission-type  values  to 
the  OT&E  test  director.  Scheduling  availability  data  are  included  in 
the  final  report.  Figure  2-13  provides  a  sample  format  for  reporting 
those  data. 


Appendix  C  contains  the  computational 
bility  determinations. 


formulas  for  all  avail  a- 


MOE 

THRESHOLD 

standard 

GOAL 

85% 

95% 

96% 

Asl 

85% 

90% 

96% 

As2 

85% 

90% 

96% 

F i gure  2-1 1 . 

Avail abi 1 i ty 

eval uation 

cri teri a  ( typical ) 

TOTAL 

TEST 

MATURE 

SYSTEM 

THRESH¬ 

OLD 

STAN¬ 
DARD  GOAL 

93.7% 

98.0% 

85% 

95%  96% 

Subsystems  A, ; 

Computing  system 

96.5% 

98.9% 

Visual  system 

98.7% 

98.9% 

Motion  system 

99.9% 

99.9% 

Mi  seel  1 aneous 

98.0% 

99.1% 

Figure  2-12.  Example  format  for  reporting  inherent  availability  ( A^- ) . 

Hypothetical  data  show  both  total  test  and  mature  system 
data  (see  phase  of  test  considerations). 


TOTAL 
__  TEST 

MATURE 

__S_YS2yi 

THRESH- 
_ qi^ _ 

STAN¬ 

DARD 

GOAL 

Asl 

87% 

91% 

85% 

90% 

96% 

As2 

86% 

90% 

85% 

90% 

95% 

Figure  2-13.  Example  foniiat  for  reporting  scheduling  availability 
(Ac)-  Hypothetical  data  show  both  total  test  and 
mature  system  values  (sec  phase  of  test  considerations). 


vs.  Ac  Implications.  There  is  a  particular  importance  to  the 
relationship  Between  A^j  and  As:  Ai  should  always  be  greater  than  As. 
The  difference  between  these  two  availabilities  provides  an  index  of 
the  availability  decrement  directly  relatable  to  possible  logistics 
and/or  admini strative  proElems.  Thus,  the  magnitude  of  im s  dif¬ 
ference  can  provide  the  user  with  an  indication  of  the  availability 
gain  to  be  derived  from  improved  logistics  support,  improved  main¬ 
tenance  management,  or  more  effective  operations  scheduling. 

Mission  Capable  Rates 

MCR  assessments  are  based  on  data  gathered  from  all  periods  of 
operational  mission  testing.  During  these  periods,  the  OLE  records 
all  clockhours  that  the  simulator  is  "possessed"^  plus  clockhours  that 
the  simulator  is  fully  mission  capable,  partially  mission  capable,  or 
nonmission  capable.  The  OLE  uses  a  mission  capable  chart  to  collect 
the  necessary  data.  This  chart  provides  a  chronol  ogical  accounting  of 
the  status  of  the  device,  with  regard  to  its  capability  to  support  its 
intended  mission.  (See  Appendix  C  for  a  sample  mission  capable  chart 
and  guidelines  for  its  completion.) 

Full  mission  capable  (FMC)  is  defined  in  AFR  65-110^  as  the  per¬ 
centage  of  possessed  time  that  a  system  is  capable  of  performing  all 
of  its  assigned  mission.  A  discrepancy  which  does  not  detract  from  or 
degrade  mission  capability  is  not  reflected  as  non-FNC  time. 

Non-FMC  time  is  divided  into  two  categories  in  accord  with  AFR 
65-110,  depending  on  its  mission  impact.  The  two  major  categories 
are:  not  mission  capable  (NMC)  and  partial  mission  capable  (PMC). 

These  two  categories  are  further  classified  as  follows: 

•  NMCM  scheduled:  This  status  occurs  whenever  the  simulator  is 
undergoing  inspections  or  preventive  maintenance  and  the  simu¬ 
lator  is  not  usable  for  mission  accomplishment.  Daily  inspec¬ 
tion,  such  as  pre-  and  post-mission  checks,  will  not  be 
counted  as  NMCM  scheduled  time.  (These  checks  are  RC 
functions. ) 

t  NMCM  unscheduled:  This  status  occurs  whenever  the  simulator 
requires  unscheduled  maintenance  which  must  be  accomplished 
before  any  further  operational  training  can  be  accomplished. 


"Possessed  time"  is  defined  as  the  time  from  initiation  of 
mission  testing  until  its  completion.  If  the  simulator  stops  oper¬ 
ating  for  any  reason  other  than  for  maintenance  or  supply,  that  time 
will  not  be  included. 
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AFR  65-110,  "Standard  Aerospace  Vehicle  and  Equipment  Inventory, 
Status,  and  Utiliaation  Reporting." 


•  MMCS:  This  status  occurs  whenever  the  sirnulato’"  is  not 

capable  of  performing  any  operational  missions  because  of  a 
lack  of  parts.  For  the  purposes  of  this  objective,  WCS  time 
under  0,5  hours  will  normally  be  reported  as  NMCM  unscheduled 
if  the  required  part  is  obtained  within  that  0.5  hours  (NMCS 
status  begins  at  the  time  the  part  is  determined  to  be  non- 
available  and  that  no  further  maintenance  can  be  accomplished). 
If  the  required  part  is  obtained  within  0.5  hours,  the  simula¬ 
tor  will  be  considered  NMCS  from  the  time  the  part  was  required 
and  maintenance  ceased. 

•  status  occurs  when  the  simulator  can  be  used  for 

operational  training  but  i t  cannot  perform  al 1  requi red 

missions  because  of  one  or  more  systems  or  subsystems  being 
inoperative.  Additionally,  maintenance  must  be  in  progress  or 
deferred  for  reasons  other  than  1  ack  of  parts  or  suppl ies. 
Daily  or  local  inspections  will  count  as  PMCM. 

•  PMCS.  This  status  occurs  when  the  simulator  can  be  used  for 

operational  training  but  it  cannot  perform  all  required 

missions  because  of  a  lack  of  parts.  The  same  criteria  as 
described  in  the  NMCS  paragraph  apply  here. 

•  Flyabl e.  This  is  the  sum  of  the  PMCS,  PMCM,  and  FMC  times. 

Using  the  data  on  clockhours  per  type  of  status,  and  the  possessed 
clockhours  data,  the  DIE  calculates  the  mission  capable  rates  using 
the  computational  formulas  contained  in  Appendix  C.  The  DLE  then  ana¬ 
lyzes  the  cause  of  less  than  desired  PMC/NMC  rates  and  determines 
what,  if  any,  specific  subsystems/ components  are  driving  these  rates. 
For  rates  which  fall  below  threshold  value,  the  DLE  determines  the 
primary  causes  and  then  examines  the  basic  reliability  and  main¬ 
tainability  data  to  obtain  additional  material  for  possible  submittal 
of  a  service  report.  Service  reports  may  be  submitted  against  speci¬ 
fic  components  if  it  is  possible  to  determine  which  components  are 
consistently  causing  the  PMC/NMC  rate. 

The  DLE  ensures  that  the  status  chart  is  updated  daily.  Addi¬ 
tionally,  he  provides  the  test  director  the  overall  system  availabil¬ 
ity  figures  for  inclusion  in  the  final  OTAE  test  report  using  an 
appropriate  reporting  table.  The  DLE  also  includes  an  evaluation  of 
work  unit  code  versus  PMC/NMC  rates  in  the  final  report.  An  example 
of  MCR  final  report  table  is  shown  in  Figure  2-14. 


TOTAL MATURE  THRESH-  STAN- 


TEST 

SYSTEM 

OLD 

DARD 

GOAL 

NMCM  scheduled 

1.2% 

1.2% 

4.0% 

1.5% 

1.2% 

NMCM  unscheduled 

7.2% 

1,6% 

9,6% 

2.5% 

2-0% 

NMCM  (total) 

8.4% 

2.8% 

13.6% 

4.0% 

3.2% 

NMCS 

1.9% 

0.0% 

3.0% 

1.0% 

0.8% 

PMCM 

16.4% 

23.2% 

PMCS 

46.3% 

32,6% 

EMC 

26.9% 

41.4% 

Flyable  total 
(PMCM+PMCS+FMC) 

89.6% 

97.2% 

83.0% 

95.0% 

96.0% 

Figure  2-14.  Example  format  for  reporting  mission  capable  rates 
(MCR).  Hypothetical  data  show  both  total  test  and 
mature  system  values  (see  phase  of  test  considerations). 
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0.  LOGISTICS  SUPPORTABILITY 


LOGISTICS  SUPPORTABILITY  EVALUATION  ELEMENTS 

Logistics  supportabil ity  concerns  those  areas  having  to  do  with 
supporting/maintaining  the  prime  system  equipment  in  its  intended 
operati ng  environment.  In  ATO  OTSE  it  is  critical  to  evaluate  the 
adequacy  of  all  logistics  support  elements  in  order  to  identify  those 
areas  of  concern  relative  to  the  support  of  the  system  throughout  its 
programmed  life-cycle.  The  consequences  of  improper  attention  to 
system  logistics  needs  can  be  costly  delays  in  operational  maintenance 
and  even  costlier  delays  and/or  interruptions  to  the  ongoing  training 
of  aircrew  personnel  . 


LOGISTICS  SUPPORTABILITY  EVALUATION  METHODS 

The  procedures  for  addressing  logistics  supportabil i ty  concerns 
are  not  currently  as  well  defined  as  are  those  for  reliability,  main¬ 
tainability,  and  availability.  Logistics  supportabil i ty  evaluations 
during  ATD  OTSE  are  mostly  qualitative  in  nature  and  are,  therefore, 
highly  dependent  on  the  expertise  and  experience  of  the  responsible 
OLE  team  personnel .  As  a  consequence,  the  results  of  such  evaluations 
often  cannot  readily  be  subjected  to  rigorous  threshold  accept/rej ect 
cri teria . 

The  factors  examined  during  logistics  supportab il i ty  evaluations 
include  the  following:  Personnel;  Support  Equioment;  Supply  Support; 
Training;  Technical  Data;  Facility;  Transporta’tion  and  Handling;  and 
Depot  Supportabil  ity  (as  applicable).  Evaluation  of  these  factors  is 
discussed  below. 

Personnel 

The  purpose  of  examining  personnel  requirements  factors  during 
ATD  OT&E  is  to  validate  and  update  the  accuracy  of  earlier  manpower 
planning  estimates  (both  numbers  and  skill  levels).  Of  interest  are 
the  personnel  required  for  the  maintenance  of  the  ATD  and  its  asso¬ 
ciated  support  equipment  throughout  its  programmed  life-cycle. 
Evaluation  of  manpower  requirements  is  conducted  to  determine  whether 
they  are  adequate  to  meet  the  requirements  specified  for  the  ATD  by 
the  MA.ICOM  operational  concept.  The  DLL's  assessments  during  OT&E  are 
compared  with  the  contractor's  maintenance  manning  proposal  and  the 
proposed  unit  manning  renuirements  (llMR)  to  identify  any  needed  modi¬ 
fications. 

There  are  two  bas'iC  tettm  :  to  estimate  manpower  require¬ 
ments.  The  primary  method  ;  •  01  systems  is  described  in  AFM 


26-3,  "Air  Force  Manpower  Standards."  This  manual  is  usually  supple¬ 
mented  by  additional  guidance  from  using  commands  to  cover  command- 
unique  manpower  standards.  The  second  method  is  the  logistics 
composite  model  (LCOM).  Although  the  LOOM  is  the  preferred  method  for 
performing  manpower  analyses  and  assessments  for  aircraft  systems,  AFM 
26-3  procedures  are  customarily  used  for  ATDs.  AFM  26-3  provides  cri¬ 
teria  and  equations  for  calculating  manpower  requirements  for  vir¬ 
tually  every  organizational  element  authorized  in  any  Air  Force  unit 
and  also  provides  rules  appropriate  for  ATD  operations  where  special 
manning  requirements  exist. 

Manpower  requirements  evaluations  include  both  direct  and 
indirect  manning  needs  by  position  and  shift.  For  ATDs,  operations 
and  maintenance  manpower  is  based  on  "position  manning  requirements," 
because  one  or  more  maintenance  personnel  must  always  be  present  in 
the  operations  or  maintenance  areas  regardless  of  the  productive  time 
expended.  This  changes  to  a  requirement  thct  tv.'o  or  more  people  be 
present  whenever  simulator  power  is  applied  (one  of  these  personnel 
must  be  a  Training  Devices  Technician  [AFSC  3417X]  qualified  in  system 
operation).  The  minimum  numbers  of  people  required  per  position  or 
shift,  and  per  month,  are  usually  calculated  by  the  OLE  team  manpower 
specialist  as  described  in  Appendix  D. 

Evaluations  of  personnel  requirements  during  OTAE  serve  two  pur¬ 
poses.  They  provide  a  basis  for  (a)  validating  cost  estimates,  and 
( b)  for  finalizing  the  device  operations  and  maintenance  manpower 
requirements. 

ATD  Support  Equipment 


ATD  support  equipment  (SE)  consists  of  all  special  tools,  moni¬ 
toring  and  checkout  equipment,  measurenent  and  calibration  equipment, 
maintenance  stands,  and  handling  equipment  required  to  support  sche¬ 
duled  and  unscheduled  maintenance  actions  associated  with  the  prime 
equipment.  SE  is  considered  "standard"  if  it  is  off-the-shelf  and/or 
already  nationally  stocklisted.  It  is  considered  "ATD  peculiar"  if  it 
is  newly  designed  and  unique  to  the  ATD  being  evaluated.  The  proce¬ 
dures,  definitions,  and  criteria  described  earlier  in  ATD  reliability 
and  maintainability  determinations  are  also  used  for  comparable  SE 
evaluations.  The  OLE  usually  uses  the  same  data  gathering  instruments 
(e.g.,  AFTO  349)  for  SE  suitability  assessments  as  were  used  during 
reliability/maintainability  evaluations.  He  may  also  etiiploy  a  special 
SE  evaluation  checklist  for  the  compilation  of  qual i tati ve  SE  infor¬ 
mation.  An  example  of  such  a  checklist  is  provided  in  Appendix  D. 

Supply  Support 

Supply  support  (Spares  and  Repair  Parts)  consists  of  all  repair¬ 
able  and  nonrepairabl  e  spares  (units,  assemblies,  modules,  etc.). 
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repair  parts,  consumables,  special  supplies,  and  related  inventories 
needed  to  support  scheduled  and  unscheduled  maintenance  actions  asso¬ 
ciated  with  the  prime  equipment,  test  and  support  equipment,  facili¬ 
ties,  and  training  equipment.  Supply  support  considerations  address 
each  maintenance  level  (echelon)  and  each  geographical  location  where 
spare/repair  parts  are  distributed  and  stocked,  the  distances  between 
stockage  points,  and  the  methods  of  material  distribution.  The  pur¬ 
pose  of  evaluating  supply  support  is  to  anticipate,  insofar  as  possi¬ 
ble,  utilization  problems  which  may  be  encountered  due  to  supply 
shortages.  A  number  of  data  sources  are  available  for  the  assessment 
of  ATD  supply  support.  These  data  sources  include  supply  consumption 
data,  condemnation  events  and  duration  of  status,  projected  support 
requirements  (provisioning),  proposed  bench  stock,  LSA  (logistics  sup¬ 
port  analysis)  reports,  availability  (NCS)  rates,  packaging  and 
handling  information,  and  service  reports  (SRs). 

The  DIE  logistics  specialist  and  supply  analyst  review  and  eval¬ 
uate  stock  usage  during  0T4E  t*"  determine  adequacy,  completeness, 
and/or  deficiencies  in  the  contrattor ' s  proposed  spares  provisioning. 
LSA  reports  also  are  reviewed  and  compared  to  actual  failure  data. 
The  OLE  determines  whether  the  contractor's  stockage  level  for  the 
device  is  acceptable. 

If  the  test  is  being  conducted  under  the  standard  Air  Force  main¬ 
tenance  data  collection  (MDC)  system,  e.g.,  AFTO  349,  then  available 
data  are  used  to  compute  actual  spares  consumption,  not-repairabl e- 
this-station  rates  (NRTS),  depot  overhaul  turnaround  times,  mean-time- 
between-demand  rates  (MTBD),  condemnation  rates,  and  cannibal ization 
rates.  These  rates  allow  the  OLE  logistics  specialist  to  recommend 
adjustments  in  spare  parts  levels  to  compensate  for  actual  high  and/or 
very  low  usage  rate  items.  Note  that  the  MDC  system  requires  that 
detailed  work  unit  codes  are  available  at  the  beginning  of  test. 

The  adequacy  of  packaging  and  handling  procedures  and  materials  is 
determined  subjectively.  Reports  are  submitted  on  an  exception  basis 
whenever  improper  packaging  and  handling  procedures  or  material  are 
discovered. 

Training 


In  addition  to  personnel  requirement  determinations,  the  adequacy 
of  maintenance  training  plans  and  programs  must  be  assessed.  To 
accomplish  this  evaluation,  which  by  necessity  is  largely  subjective 
in  nature,  the  OLE,  with  the  assistance  of  ATC  training  specialists, 
reviews  the  planned  training  to  determine  area  of  potential  problems. 
This  review  of  training  plans  and  proposed  course  oulines  is  accom¬ 
plished  by  the  OLE  training  specialist,  and  any  observed  training 
inadequacies  are  reported  to  the  OTAE  test  director.  The  results  of 
this  training  evaluation  may  modify  subsequent  ATC  course  content,  OJT 
procedures,  training  aids,  and  associated  training  methods. 


Technical  Data 


Technical  data  are  the  TOs,  drawings,  microfilm,  operating  and 
maintenance  instructions,  modification  instructions,  provisioning  and 
facilities  informatioi,  specifications,  inspection  and  calibration 
procedures,  and  computer  programs  required  to  support  installation, 
checkout,  operation,  and  maintenance  of  the  prime  equipment  and  asso¬ 
ciated  test  and  support  equipment. 

DLE  techm^al  data  specialists/simulator  technicians  conduct  pre¬ 
liminary  evaluations  of  technical  data  during  TO  reviews,  the  main¬ 
tainability  demonstration,  and  during  whatever  other  times  the 
appropriate  technical  data  are  available.  All  utilized  technical 
data,  including  contractor  drawings,  arc  evaluated  for  suitability, 
adequacy,  completeness,  and  correctness.  The  evaluations  provide 
identification  of  unsatisfactory  maintenance  procedures  in  technical 
data;  i denti fication  of  inconsistencies  with  general  hardware  TOs; 
assurance  that  all  safety  requi  rcinents  are  incl  uded  in  the  handbooks 
and  that  warning  and  caution  notes  have  been  incorporated;  assurance 
that  the  -6  handbook  reflects  repair  restrictions  and  time-change 
requi remen ts ;  and  analysis  of  bencb-check-serviceabl e  rates  and  could- 
not-dupl icate  rates  to  identify  those  occurrences  caused  or  encouraged 
by  poor  technical  data. 

The  availability  of  technical  data  (or  lack  thereof)  during  early 
OT-^E  phases  sometimes  poses  a  problem.  In  some  cases,  evaluation  of 
technical  data  may  have  to  be  postponed  until  sufficient  such  data  are 
avail  able. 

An  in-depth  comprehensive  analysis  of  technical  data  during  AID 
OT&E  o^'ten  is  not  possible  since  a  complete  set  of  verified  and  vali- 
datec  technical  data  will  not  normally  be  available  for  review.  Tiie 
use  of  overly  detailed  procedures  for  evaluation  of  technical  data  is 
therefore  usually  inappropriate  during  ATD  lOT&E.  Even  the  evaluation 
checklists  included  in  Appendix  D  may  be  too  extensive  for  use  with 
early  technical  data  and  may  necessitate  an  approach  wherein  only 

glaring  d^’ scr-q  anc  ies  can  be  noted. 

f  i  c  U‘M.:  1  e  s  in  preliminary  technical  publications  are  identified 
using  AF  TO  Fcr-m  ISd.  The  DLF  fechniral  data  specialist  will  ensur- 

that  Topics  of  A)  TO  Form  158  and  supporting  review  coi.rients  are  made 
.un'iiblo  for  this  evaluation.  formally,  a  more  comprehensive  tech¬ 
nic. j1  data  review  is  possible  during  FOT^E  subsequent  to  verification 

and  validation.  In  this  case,  Af  TO  Form  72  is  used  instead  of  Form 

I'/-,  tu  record  comments  and  fief  ic  iencios. 

to  aid  in  analysis  and  evaluation  of  pertinent  technical  data, 
checklists  can  be  rmpl  oy('d  that  a.Mress  manual  content  and  style. 

,)f  technical  datv=i  evaluation  checklists  are  contained  in 


Faci 1 1  ties 


Facilities  consist  of  the  physical  plant,  real  portable 

buildings,  housing,  intermediate  shops,  depots,  etc.,  rroui’^ed  to  sup¬ 
port  the  operational  and  maintenance  functions  associated  with  the 
AID,  its  test  and  support  equipment,  training  equi:”'ent  required 
throughout  the  ATD's  life-cycle,  storage  for  scare ■  pa i parts  and 
data,  admini  strati  ve  space  for  operator  mai  ntena^'ce  pe^'sonnel  ,  and 
training  operations  areas. 

The  adequacy  of  AID  support  facilities  is  vn  e'(^a  of  0T^.E 
assessment  that  depends  largely  on  subjective  The  exper¬ 

tise  and  experience  of  the  responsible  RLE  persre,nel  tcerLtore  greatly 
affect  the  outcomes  of  the  evaluation.  All  ''v:n  n  activities 

should  be  monitored  to  identify  any  facilities  '“equi  re-  ts  that  are 
not  adequately  met.  OLE  personnel  st*ould  review  applicdMe  publica¬ 
tions  and  maintenance  and  SE  requirements,  and  report  any  required  new 
facilities,  additions,  or  modif icatiens  deemed  nocessar'y  to  support 
the  ATD.  Support  can  also  be  solicited  from  other  resp'^.sible  agen¬ 
cies,  including  the  maintenance  contractor,  S'niSRO,  and  ATC, 

in  acccunpl  ishing  this  evaluation. 

The  RLE  facilities  specialist  customarily  evaluates  the  facili¬ 
ties  using  a  facilities  evaluation  checklist.  -roMem  areas  are 
reported  on  an  exception  basis;  i.e.,  if  a  proole^'  or  potential 
problem  is  identi  fif'd,  it  is  repof'teo  usinq  the  U  reporting 

syst-^m.  Quantitative  data  are  kept  on  thosc'  i  i  l  i  ty-rel  ited  systems 
which  are  integral  to  the  simulator,  rega*  Hess  of  wlother  these 
systems  are  managed  as  equipment  or  real  property  . 

Tne  facilities  eVv^iuation  usually  en^_  onqoi  ^  ses  a  ^ho»'.’<ugh  subjec¬ 
tive  evaluation  of  the  maintenance  wiark  areas,  clasb‘‘:om  training 
areas,  brief  i  ng/debriefi  ng  areas,  storage  araas,  supr'rvision  areas, 
computer  areas,  hydraulic  pump  room  (motion-t'a  eci  ATRs),  simulator 
bay,  computer  bay,  i  nstriic  tor-operator  station,  etc.,  using  a 
checklist  tha;t  allows,  as  a  minimum,  evaluatiar  v'*  the  adequacy  of  the 
f ol  1 owi ng  : 

( 1 )  Space 

(2)  Electrical  power  systems 

(3)  Lighting 

(4)  Cooling  systems 

(5)  Simulator  clearances 

( 6 )  Conven ience  fac  tors 


(7)  Emergency  exits 

(8)  Quality  of  materials  used 

(9)  Human  factors  (related  to  facilities) 

(10)  Storage  requi remcnts 

(11)  Built-in  support  equipment 

(12)  Fi re  ex  ting ui shi ng/ suppress  ion  systems 

(13)  External  ingress/egress 

(14)  Security  (physical  and  classification) 

The  OLE  analyzes  and  evaluates  the  facilities  evaluation  check¬ 
lists  completed  by  the  various  evaluators  for  their  area  of  interest. 
The  OLE  may  initiate  service  reports  at  any  time  a  facilities  problem 
is  detected.  The  OLE  summarizes  facilities  problems  in  interim  and 
final  reports  to  the  test  director.  A  typical  facilities  evaluation 
checHist  is  provided  in  Appendix  D. 

Transporta tion  and  Hand! ing 

Transportation  and  handling  addresses  those  special  previsions 
such  as  reusable  containers  and  supplies  necessary  to  support 
packaging,  preservation,  storage,  handling,  and/or  transportation  of 
prime  equip^nent,  test  and  support  equipment,  soare/repair  parts,  tech¬ 
nic  al  data  ,  and  fac il i ti es . 

Logistics  evaluation  personnel  observe  contractor  transportati on 
and  handling  of  the  AID  equipment  and  supplies.  Additionally,  the 
transportation  design  characteristics  of  all  major  components  are 
reviewed.  As  problems  or  potential  problems  are  detected,  they  are 
reported  by  the  SR  process  to  the  OLE.  The  transporta  tion  and 
handling  checklist  is  completed  for  the  transport  of  the  simulator/ 
systr>n^^  and  checklists  will  be  completed  on  a  sampling  of  varnous 
sunp^  if.^s .  All  bindings  are  included  in  the  final  OT-^E  report.  An 
c^a^iple  of  a  transportation  and  handling  checklist  is  shown  ^n 
Appendix  D. 

^-^il  standard  tr<jnsportation  and  handling  evaluations  during  AT^ 

nay  not  be  possible  to  accomplish  if  military  standard  transpor¬ 
tation  and  handling  requi  retnents  are  waived,  for  example,  allowing  the 
contractor  to  use  "best  co'’imercial  practice"  procedures.  Such  a 
waiver  is  a  coi’nnon  occurrence. 


^PP0>^tabi  1  i  ty  (  as  appT  icable) 

Depot  stjpportdbn  ity  is  concerned  with  the  projected  workload, 
the  skills  and  manpower  required  for  repair,  facility  requirements, 
tools  o:id  test  equipment,  software,  data,  training,  and  spare  parts 
reqjM.'ed  to  develop  an  organic  depot  overhaul  capability. 

A  number  of  data  sources  may  be  utilized  in  evaluation  of  depot 
Mjpportdbil  ity .  These  include  the  depot  facility  site  survey,  tech¬ 
nical  data  and  drawings,  repairable  item  lists,  automated  test  equip¬ 
ment,  software  documentation,  training  plan/course  identification, 
service  reports,  logistics  support  analysis  (LSA  reports),  firmware 
documentation  DIDs  (Data  Item  Descriptions),  and  tools  and  test  equip¬ 
ment  (both  peculiar  and  common). 

An  experienced  simulator  logistics  specialist  is  required  for  the 
eva'^uation  of  depot  supportabil  i ty ,  because  relevant  data  must  be  con¬ 
sidered  subjectively.  Findings  in  this  area  are  compiled  into  a  depot 
maintenance  capability  development  plan  by  the  DIE  and  provided  to  the 
test  director  for  incorporation  into  the  final  report. 

LOGISTICS  SUPPORTARILITY:  TEST  DIRECTOR  CONCERNS 
Phase  of  Test  Considerations 


Logistics  supportabil ity  assessments  provide  insight  into  other 
suitability  factors  and  impact  a  number  of  critical  decision  areas. 
The  most  apparent  of  these  has  to  do  with  the  planning  and  budgeting 
for  downstream  life-cycle  costs.  Another  key  area  concerns  the 
logistics  support  complement  of  future  devices  of  the  same  or  similar 
type  and  application.  For  example,  support  equipment  may  have  been 
specified  for  the  device  under  test  that  is  not  really  needed,  and 
conversely,  necessary  SE  may  have  been  omitted-  This  finding  could  be 
used  to  define  future  system  support  packages. 

P^ersonn^l  Requirements 

Logisitics  supnortabil  ity  assessments  ordinarily  will  be  carried 
out  by  delegating  such  responsibil  i  ties  to  available  simulator  tech¬ 
nicians.  However,  contact  by  the  tost  director  with  expert  personnel 
in  the  following  identified  areas  of  interest  is  strongly  recomnendod 
during  early  test  planning  and  for  review  both  of  planned  tost  proce¬ 
dures  and  the  logistiv_s  supportabil  ity  portion  of  the  final  report. 
In  this  way,  these  personnel  need  only  bo  accessed  for  comparati vel y 
brief  periods  during  conduct  of  the  test.  Accessing  Liusr  two  spe¬ 
cialists  is  of  particular  importance. 
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A 


SOURCE 

AFSC 

NUMHLK 

Simulator  Te c li n i c  i an 

AFLC 

3al7X* 

- 

Su pp 1 y  Sy s terns  Specialist 

AFLC 

bb'l 

1 

Manpower  Management  Oiticer 

MAJCOM 

7424 

1 

Supply  Operations  Officer 

MAJCOM 

b  4  2 '4 

1 

Training  Technic ian 

MAJCOM/ATC 

73172 

i 

Transportation  Officer 

AFLC 

b034 

1 

civil  Engineering  Officer 

OOALC/MAJCOM 

331b 

1 

Packaging  Specialist 

AFLC 

b02‘:»2 

1 

*’'X"  Specialty  area  as  appropriate  to  device  uiui-jr  evaluation. 


E.  OPERATING  AND  SUPPORT  COSTS 


O&S  COST  EVALUATION  ELEMENTS 

Six  major  elements  of  ATD  O&S  costs  should  be  considered  during 
OT&E.  These  are:  (1)  simulator  maintenance  manpower,  (2)  replenish¬ 
ment  spares,  (3)  simulator  maintenance  materiel,  (4)  support  equip¬ 
ment,  (5)  facility  maintenance  costs,  and  (6)  electrical  power  costs. 

Simulator  Maintenance  Manpower 


This  cost  element  refers  to  the  cost  of  manpower  required  to 
maintain  the  simulator  in  its  intended  operational  environment.  This 
element  is  basically  the  cost  of  providing  those  personnel  needed  to 
meet  the  base- level  maintenance  requirements  of  the  simulator.  This 
element  includes  all  manpower  costs  incurred  to  meet  the  direct  main¬ 
tenance  demands  of  the  simulator,  to  provide  for  maintenance  super¬ 
vision,  and  to  cover  administrative  requirements  such  as  leave, 
sickness,  TOY,  etc.  Included  are  personnel  at  both  the  organization 
and  intermediate  levels.  Not  included,  however,  are  depot  level  main¬ 
tenance  personnel  who  may  be  required  periodically  at  centralized 
depot  repair  facilities.  If  contractor  field  service  support  (CFS) 
aiui/ur  field  service  representati ves  (FSR)  are  required,  such  costs  are 
also  incorporated. 

Replenishment  Spares 


This  element  covers  the  cost  of  procuring  system  assemblies, 
spares,  and  repairable  parts  which  are  normally  repaired  and  returned 
to  stock.  In  addition,  it  includes  procurement  of  stock  levels  that 
are  not  provided  by  initial  spares  procurement.  These  are  centrally 
managed  investment  type  items. 

Simulator  Maintenance  Materiel 

This  element  is  the  cost  of  purchasing  materiel  from  the  general 
and  system  support  division  of  the  stock  fund.  This  includes  all 
nonrepa i rabl e  expense-type  items  including  bench  stock,  direct 
materiel,  and  base  operating  consumables  used  in  the  organizational 
and  intermediate  maintenance  activities  at  base  level. 

Support  Equipment 

This  clement  covers  the  cost  of  procuring  common  maintenance  and 
repair  shop  equipment,  instruments,  test  equipment,  and  spares  for 
this  equipment.  These  equipment  demands  are  generated  by  a  need  to 
(1)  replace  peculiar  support  equipment  bought  using  system  prncurerent 
funds,  (2)  obtain  common,  off-the-shelf  support  ogu'p'V'nt  tiiat  is 


needed  to  support  operations  as  production  systems  in  the  operating 
inventory,  and  (3)  provide  replenishment  of  common  equipment  that  is 
no  longer  repairable. 

Simulator  Facility 


This  cost  element  includes  all  direct  labor,  materiel  overhead, 
and  other  direct  charges  incurred  in  maintenance  of  the  simulator 
facility  (it  includes  maintenance  of  real  property  where  applicable). 

Electrical  Power 


This  cost  element  reflects  the  annual  cost  of  batv,  j,  generator, 
and  commercially  supplied  power  for  the  operation  of  the  simulator. 


O&S  COSTS:  TEST  DIRECTOR  CONCERNS 
Phase  of  Test  Considerations 


A  cost  analyst  should  be  involved  in  early  planning  meetings  to 
identify  the  specific  cost- related  data  it  will  be  necessary  to  track 
during  test.  At  this  point,  the  analyst  may  be  needed  only  for  a  few 
weeks.  As  the  test  proceeds,  the  relevant  data  are  made  available  to 
the  cost  analyst  who  will  effect  the  necessary  cost  calculations  and 
provide  the  appropriate  reports  to  the  DIE. 

A  number  of  factors  will  affect  OSS  cost  estimates.  Support- 
ability  factors  are  of  particular  impact,  for  example.  If  the  sup¬ 
porting  technical  data  are  inadequate,  the  skill  levels  of  technical 
personnel  will  have  to  be  increased  to  compensate  for  that  inadequacy. 
On  the  other  hand,  if  the  technical  data  are  of  high  quality,  lower 
skill  level  personnel  can  be  utilized,  thus  decreasing  the  support 
costs  for  ATD  personnel . 

Initial  O&S  cost  estimates  from  early  OTSE  phases  can  be  used  to 
develop  inputs  to  the  using  command's  OXM  budget.  Also,  such  O^IS  cost 
data  can  be  used  to  support  a  source  selection  decision.  For  example, 
should  the  cost  of  electrical  power  to  run  device  A  be  substantially 
greater  than  that  required  to  run  device  B,  then,  with  other  factors 
being  equal,  a  decision  to  buy  device  B  could  be  justified. 

Personnel  Requirements 


As  noted  above,  a  cost  analyst  is  needed  to  specify  needed  infor¬ 
mation  early  on,  and  to  perform  the  necessary  analyses  after  those 
data  are  collected. 

SPECIALTY  SOURCE  AFSC  NUMBER 


Cost  Analyst 


MAJCOM 


6746 


1 


CHAPTER  3 


SOFTWARE  SUITABILITY 


INTRODUCTION 

Modern  ATDs  have  a  substantial  software  component.  Software 
controlled  elements  of  ATDs  normally  provide  the  “flying"  characteris¬ 
tics  of  the  device  via  programmed  aerodynamic  models,  as  well  as  the 
operating  characteristics  of  navigational  ,  weapons,  ECM,  and  communi¬ 
cations  systems,  among  others.  In  addition,  ATD  training  support 
features  (freeze,  record/ repl ay ,  etc.)  are  controlled  largely  by  soft¬ 
ware. 


The  extensive  role  of  software  in  ATD  system  operation  creates  a 
substantial  need  for  its  critical  evaluation  during  OTAE.  The  proce¬ 
dures  for  that  evaluation  are  quite  different  from  those  appropriate 
for  hardware  suitability  assessments,  however,  because  there  are 
distinct  differences  between  hardware  and  software  failure  effects 
which  must  be  recognized.  Software  evaluation  procedures  must  reflect 
those  differences. 

Important  to  understanding  the  concepts  of  hardware/ software 
testing  during  ATD  OT&E  is  a  knowledge  of  the  difference  between  hard¬ 
ware  and  software  failure  effects.  Ha'^dware  failures  are  almost 
always  the  result  of  component  damage  or  .k teriorati on  due  to  age, 
humidity,  temperature,  vibration,  etc.  hardware  failures  will  recur. 
Software  "failures"  arise  only  from  program  design  and/or  implemen¬ 
tation  errors.  Software  does  not  fail  or  degrade  over  time.  The 
occurrence  of  a  system  failure  due  to  a  software  failure  may  be  simi¬ 
lar,  in  net  effect,  to  a  hardware  failure.  However,  once  the  software 
has  been  corrected,  it  will  never  "fail"  in  the  same  way  again.  As  a 
consequence,  the  concepts,  measures,  and  techniques  appropriate  for 
hardware  suitability  evaluation  cannot  be  used  directly  to  test  soft¬ 
ware. 


RESPONSIBILITIES  OF  SOFTWARE  EVALUATION  PERSONNEL 
Dj?  p  u  ty^f  0  f^v^d  r  E  V  a  1  u  a  t  i  0  n 

The  focal  point  for  all  software  evaluation  matters  is  the  Deputy 
for  Software  Evaluation  (OSE).  Specifically,  the  DSE: 

(a)  f^anages  the  software  evaluators.  This  includes  planning, 
schodul  ing,  and  coordinating  oval  uation  activities  and 
assigning  eval uators  to  perform  regui red  tasks . 
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(b)  Establishes  any  unique  procedures  required  for  effective- 
control  of  software  related  activities. 

(c)  Coordinates  software  activities  with  other  test  activities 
and  refers  potential  schedule  or  resource  conflicts  to  the 
OT&E  test  director  for  resolution. 

( d)  Prepares  and  submits  status  reports,  as  required,  to  the  test 
di  rector . 

(e)  Participates  in  the  software  configuration  control  process. 
Maintains  cognizance  of  all  software  changes  proposed  and  in 
various  stages  of  implementation.  Chairs  a  software  problem 
review  board  during  OTSE. 

Software  Evaluators  and  Analysts 


Under  the  guidance  of  the  DSE,  the  evaluators  are  responsible  for 
making  a  unified  assessment  of  the  software.  The  specific  respon¬ 
sibilities  of  these  software  evaluation  personnel  supporting  the  OT^f 
are ; 

(a)  Assist  the  DSE  in  selecting  software  documentation  and 
source  listing  to  be  evaluated. 

( b)  Assist  the  DSE  in  preparation  of  the  software  assessment 
portions  of  the  final  report. 

(c)  Assist  the  DSE  in  adr'inistering  the  Software  Operator- 
Machine  Interface  Questionnaires. 

( d)  Collect,  monitor,  and  review  data  for  coinputer  support 
resources  and  all  software  objectives. 

(e)  Identify  software  discrepancies  and  monitor  corrective 
actions . 

( f)  Complete  software  documentation  and  software  source  listing 
questionnaires,  and  opera  tor- i n ter  face  questionnaires. 

( g)  Prepare  Computer  Program  Observation  Reports  ( A1 If C  Form 
?07)  to  document  anomalies  or  problems  noted  during  software 
suitability  evaluation. 


At  TEC  ATD  SOFTWARE  [.VALUATION  APPROACH 


As  noted  in  the  first  chaptf’r  of  this  vol  (iaie ,  software  evalu.iti'n 
guidelines  have  l)een  provided  in  the  five-volu'e  Af  'm:  he'tlM'of, 


’‘Software  OT&E  Guidelines,”  AFTEC  software  OTAE  concerns  and  asso¬ 
ciated  evaluation  techniques  continue  to  evolve,  however,  and  no  set 
of  methods  has  been  developed  to  date  which  applies  to  all  systems  and 
all  OTAEs.  Therefore,  the  intent  of  the  AFTEC  handbook^  is  not  direc¬ 
tive,  but  rather  is  that  of  providing  a  source  of  infonnation  and 
guidelines  regarding  software  OT^E. 

The  AFTEC  approach  to  software  evaluation  documented  in  that 
handbook  distinguishes  between  "software  suitability,”  which  is  con¬ 
cerned  with  maintainability  and  usability  of  software,  and  "software 
effectiveness,"  which  is  concerned  with  the  performance  of  the  soft¬ 
ware  from  the  standpoint  of  system  operational  effectiveness.  The 
current  AFTEC  approach  to  software  effectiveness  per  se  considers  it 
to  be  part  of  the  total  ATD  operational  effectiveness  evaluation.  As 
a  result,  separately  defined  evaluation  procedures  for  sofware  effec¬ 
tiveness  have  not  been  developed.  However,  for  ATD  software  suitabil¬ 
ity,  AFTEC  has  developed  an  approach  based  largely  on  the  use  of 
subjective  questionnaires. 

This  chapter,  therefore,  is  primarily  directed  to  the  topic  of 
ATD  software  sui tabil i ty  evaluation,  as  prescribed  by  AFTEC ‘s  current 
software  evaluation  handbook.  Definitions  for  the  major  elements  of 
software  suitability  are  provided,  as  are  generic  personnel  require¬ 
ments  and  special  concerns  of  the  test  director  relative  to  software 
evaluation  during  ATD  OTSE. 

Elements  of  Software  Suitability 

Operational  suitability  evaluation  for  software  will  typically 
address  the  overall  concerns  of  software  maintainabil ity  and  software 
usability.  Under  these  two  categories,  a  number  of  subcloments  are 
considered  as  shown  in  Figure  3-1, 

Each  of  these  two  major  ele^nents  is  defined  more  fully  in  the 
following  subsections:  A.  Software  Maintainabil  ity ;  and  B.  Software 
Usabil ity. 


Inasmuch  as  software  evaluation  techniques  remain  an  evolving 
process,  the  test  director  should  bo  certain  to  access  the  most  recent 
editions  of  that  handbook  available  to  him. 
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The  intent  here  is  to  acquaint  the  now  test  director  with  the 
basic  philosophy  of  software  suitability  evaluations  as  currently 
devf'lopod  at  AE'TEC.  Therefore,  much  of  the  material  in  those  sc^ctions 
has  been  excerpted  directly  from  the  Af- TEC  handfiooks  noted  above. 
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Modul  an'  ty 
Descri ptiveness 
Consi stency 
Simp!  icity 
Expandabil i ty 
Instrumentation 

SOURCE  LISTINGS 
Modul ari ty 
Descriptf veness 
Consi stency 
Simp! icity 
Expandabil i ty 
Instrumentation 

Figure  3-1.  Subelements 
usabil ity • 


OPERATOR-MACHINE  INTERFACE 
Descriptiveness 
Consi stency 
Simp! icity 
Assurabil ity 
Control  1 abil ity 
Workload  Reasonability 

FUNCTIONAL  PERFORMANCE 
Fail ures 


software  maintainabil  ity  and 


A.  SOFTWARE  MAIWTAINARILITY 


So. cware  consists  of  a  set  of  computer  instructions  and  data 
structured  into  programs,  and  the  associated  documentation  on  the 
design,  implementation,  test,  support,  and  operation  of  those 
programs.  Each  software  program  is  separately  evaluated  and  consists 
of  a  set  of  components  called  modules.  A  module  may,  in  general,  be 
at  any  conceptual  level  of  the  program.  In  FORTRAN,  modules  are 
generally  defined  to  be  subroutines;  in  COBOL,  a  module  is  usually  a 
total  program.  The  DSE  must  decide  on  the  definition  of  a  module  for 
the  specific  language  and  system  to  be  evaluated. 


SOFTWARE  MAINTAINABILITY  EVALUATION  ELEMENTS 

In  the  course  of  using  an  ATD,  as  well  as  during  its  OT^E,  it  may 
become  necessary  to  maintain  or  change  its  software.  Such  software 
changes  are  made  to:  (a)  correct  errors,  (b)  add  system  capabilities, 
(c)  delete  features  from  programs,  or  (d)  modify  software  to  be  com¬ 
patible  with  hardware  changes.  The  maintainability  of  the  software  is 
a  function  of  those  characteristics  of  the  software  and  its  computer 
support  resources  which  affect  the  ability  of  software  programner/ 
analysts  to  make  such  changes. 

The  AFTEC  methodology  for  evaluating  software  maintainability  is 
based  on  the  use  of  closed  form  questionnaires  with  optional  v^itten 
ccMnments.  These  questionnai res  are  designed  to  determine  the  presence 
or  absence  of  certain  desirable  attributes  in  a  given  software  pro¬ 
duct.  The  elements  of  software  maintainability  and  their  relation¬ 
ships,  as  shown  in  Figure  3-1,  are  described  in  following  paragraphs. 
The  hierarchical  evaluation  structure  shown  in  the  figure  enables  the 
identification  of  potential  maintainabil  i ty  problems  at  various 
levels:  category  ( documentation,  source  listings),  charac teri stic 

(modularity,  consistency,  etc.).  For  each  software  program  there  are 
two  related  categories  that  are  evaluated  for  charac teri sti cs  which 
affect  software  maintainabil i ty .  These  are  (1)  software  documen¬ 
tation,  and  (?)  software  source  listings.  A  third  category,  compiiter 
support  resources,  is  also  appropriate  for  such  evaluation.  This 
category  includes  all  the  relevant  resources  such  as  ./liMii.jry  soft¬ 
ware,  computer  equipment.  Facilities,  etc.,  which  will  be  used  to  sup¬ 
port  the  maintenance  of  the  software  being  evaluated.  However, 
procedures  for  the  evaluation  of  computer  support  resources  are 
currently  under  development  by  AFTEC  and  therefore  cannot  be  further 
addressed  in  this  present  Handbook. 

Software^  nocurnentation 

Software  program  docimientation  is  the  set  of  r(‘qui  rfMiuuUs ,  design 
spec  i  f  ica ti ons ,  gui del  ines  ,  opera tional  procedures  ,  t(*st  i nfco'mati on  , 


problem  reports,  etc.  which  in  total  form  comprise  the  written  des¬ 
cription  of  a  computer  program.  The  primary  documentation  used  in 
this  evaluation  consists  of  the  documents  containing  program  design 
specifications,  program  testing  information  and  procedures,  and 
program  maintenance  information.  These  documents  may  have  a  variety 
of  configurations  depending  upon  the  particular  application.  The 
documents  are  eval uated  both  for  content  and  for  general  physi cal 
structure  (format).  The  content  evaluation  is  primarily  concerned 
with  how  well  the  overall  program  has  been  designed  (as  documented) 
for  maintainability.  The  fonnat  evaluation  is  primarily  aimed  at  how 
the  physical  structure  of  the  documentation  (table  of  contents,  index, 
numbering  schemes,  modular  separation  of  parts,  etc.)  aids  in  under¬ 
standing  or  locating  program  information. 

Software  Source  Listings 

Software  source  listings  are  the  computer  generated  (or  equiva¬ 
lent)  form  of  the  program  code  in  its  source  language  (e.g.,  FORTRAN, 
COBOL,  JOVIAL,  AdA,  assembly  language,  etc.).  The  source  listing 
represents  the  program  as  implemented,  in  contrast  to  the  documen¬ 
tation  which  for  the  most  part  represents  the  program  design  or  iinpl  e- 
mentation  plan.  In  essence,  source  listings  are  also  considered  a 
form  of  program  documentation,  but  for  maintainability  evaluation,  a 
distinction  is  made. 

The  source  listing  evaluation  consists  of  a  separate  evaluation 
of  each  selected  module's  source  listing  and  the  consistency  between 
the  module's  source  listing  and  the  related  module  docu^nontation.  The 
separate  module  evaluations  are  summarized  to  yield  an  overall  eval¬ 
uation  of  the  software  source  listing  for  the  given  program. 

So ftware  Mai ntainabi 1 i ty  Subelements 

The  maintainability  of  software  documentation  and  source  listings 
is  determined  by  examining  six  subelements:  modularity,  descrip- 

tiveness,  consistency,  simplicity,  expandability,  and  instrumentation. 
Discussions  of  these  subelements  and  their  application  in  the  eval¬ 
uation  of  the  software  documentation  and  source  listings  are  provided 
be! ow. 

Modularity.  Software  possesses  the  charac ter i stic  of  modularity 
to  the  extent  that  a  logical  partitioning  of  software  into  parts,  con^ 
ponents,  ard/or  modules  ha*^  occurred.  Software  that  is  the  easiest  to 
understand  and  change  is  ccnposcd  of  independent  modules,  Tach  soft¬ 
ware  product  is  therefore  evdluat*^d  in  relation  to  the  extent  to  whi^h 
its  logical  parts,  components,  and  modules  are  independent.  The  fewer 
and  simpler  the  connections  between  [uirts,  the  easier  it  is  to  under¬ 
stand  each  module  without  reference  to  other  parts.  Minimizing  con¬ 
nections  between  parts  also  minimizes  the  paths  along  wliich  changes 
and  errors  can  propagate  into  other  parts  of  the  system,  thus  rediic  ing 
the  occurrence  of  side  effects  within  the  system. 


As  a  general  guideline,  modularity  implies  that  a  given  module 
consists  of  only  a  few  easily  recognizable  functions  which  are  closely 
related  and  that  a  minimal  number  of  links  exist  to  other  nodules-- 
preferably  only  via  parameters  passed  in  a  calling  parameter  list.  In 
addition,  the  physical  format  of  the  documentation  should  exhibit  com¬ 
ponent  independence  for  its  sections,  volumes,  etc.  There  should  be 
separate  sections  for  the  description  of  the  major  parts  which  a  given 
document's  purpose  encompasses. 

Descriptiveness.  Software  possesses  the  characteristic  of  des- 
criptiveness  to  the  extent  that  it  contains  Information  regarding  its 
objectives,  assumptions,  inputs,  processing,  outputs,  components, 
revision  status,  etc.  This  attribute  is  very  important  in  under¬ 
standing  software.  Documentation  should  have  a  descriptive  format  and 
contain  useful  explanations  of  the  software  program  design.  The 
objectives,  assumptions,  inputs,  etc.,  are  useful  (in  varying  degrees 
of  detail)  in  both  documentation  and  source  listings.  In  addition, 
the  descriptiveness  of  the  source  1 anguage  syntax  and  the  judicious 
use  of  source  commentary  greatly  aids  efforts  to  understand  the 
program  operation. 

Consi stency .  Software  possesses  the  characters  Stic  of  con¬ 
sistency  to  tTie  extent  the  software  products  correlate  and  contain 
uniform  notation,  terminology  and  symbology.  The  use  of  standards  in 
documentation,  flow  chart  construction  and  certain  conventions  in  I/O 
processing,  error  processing,  module  interfacing,  naming  of  modules/ 
variables,  etc.  are  typical  reflections  of  consistency.  Attention  to 
consistency  characteristics  can  greatly  aid  one  in  understanding  the 
proqram.  Consistency  allows  one  to  generalize  easily.  For  example, 
programs  using  consistent  conventions  require  that  the  format  of 
modules  be  similar.  Thus,  by  learning  the  format  of  one  module 
(preface  block,  declaration  format,  error  checks,  etc.),  the  format  of 
ail  modules  is  learned.  This  allows  one  to  concentrate  on 
understanding  the  true  complexities  of  an  algorithm,  data  structure, 
etc . 


Simpl ici ty .  Software  possesses  the  characteristic  of  simplicity 
to  the  extent  that  it  lacks  complexity  in  organi zation ,  language,  and 
implementation  techniques,  and  to  the  extent  that  it  reflects  the  use 
of  singularity  concepts  and  fundamental  structures.  The  aspects  of 
software  complexity  (or  lack  of  simplicity)  that  are  c^mphasizod  in  the 
evaluation  relate  primarily  to  the  concepts  of  size  and  primitives. 
The  less  there  is  to  discriminate  and  the  more  use  there  is  of  basic 
or  primitive  techniques,  structures,  etc.,  the  simpler  the  software 
will  tend  to  be.  The  use  of  a  high  order  language  as  opposed  to  an 
assembly  language  tends  to  make  a  program  simpler  to  understand, 
because  there  are  fewer  discriminations  which  have  to  be  made.  There 
are  certain  programming  considerations  such  as  dynamic  allocation  of 
resources  and  recursive/reentrant  coding  which  can  greatly  complicate 
the  data  and  control  flow.  Real-time  programs,  because  of  the 
reqiii  rf'inent  for  timing  constraints  and  efficiency,  tend  to  have  more 
control  compl ex i ty . 


The  sheer  bulk  of  a  module  (number  of  operators,  operands,  nested 
control  structures,  nested  data  structures,  executable  statements, 
statement  labels,  decision  parameters,  etc.)  will  determine  to  a 
great  extent  how  simple  or  complex  the  source  code  is.  While  it  is 
recognized  that  the  particular  application  itself  may  preclude  the 
possibility  of  a  reasonably  simple  design  or  implementation,  because 
of  requirements  such  as  a  particularly  complex  real-time  scheduling 
algorithm  or  high  level  mathematical  or  other  theoretical  considera¬ 
tions,  this  complexity  nonetheless  makes  maintenance  more  difficult. 

Expandabil i ty.  Software  possesses  the  characteristic  of  expand¬ 
ability  to  the  extent  that  a  physical  change  to  information,  com¬ 

putational  functions,  data  storage,  or  execution  time  can  be  easily 
accomplished  once  the  nature  of  what  is  to  be  changed  is  understood. 

Software  may  be  perfectly  understandable,  but  not  easily  expand¬ 
able.  If  the  design  of  the  program  has  not  allowed  for  a  flexible 
timing  scheme  or  a  reasonable  storage  margin,  then  even  minor  changes 
may  be  extremely  difficult  to  implement.  Parameterization  of  con¬ 
stants  and  basic  data  structure  sizes  usually  improves  expandability. 
It  is  also  very  important  that  the  documentation  include  explanations 
of  how  to  effect  increases/decreases  in  data  structure  sizes  or 

changes  to  the  timing  scheme,  and  the  limitations  of  such  program 

expandability  should  be  clear.  The  numbering  schemes  for  source 
listings,  documentation  narrative,  and  graphic  materials  must  be  care¬ 
fully  considered  so  that  physical  modifications  to  the  code  and  docu¬ 
mentation  can  be  easily  accomplished  when  necessary. 

Instrumentation.  Software  possesses  the  characteristic  of 

i nstrumentation  to  the  extent  that  it  contains  aids  which  enhance 

testing.  For  the  most  part,  the  documentation  is  evaluated  on  how 
well  the  program  has  been  designed  to  include  test  aids  (instruments), 
while  the  source  listings  are  evaluated  on  how  well  the  code  seems  to 
be  implemented  to  allow  for  testing  through  the  use  of  such  test  aids. 
This  part  of  the  evaluation  reflects  the  concern  (from  a  maintain¬ 
ability  viewpoint)  that  the  software  be  designed  and  implcmerted  so 
that  instrumentation  is  either  imbedded  within  the  program,  can  be 
easily  inserted  into  the  program,  is  available  through  a  support  soft¬ 
ware  system,  or  is  available  through  a  combination  of  these  capabili¬ 
ties. 


SOFTTJARE  MAINTAINABILITY  EVALUATION  METHODS 

The  basic  software  maintainabil ity  evaluation  procedure  involves 
four  distinct  phases:  planning,  calibration,  assessment,  and  analysis 

During  the  planning  phase,  the  test  manager  and  the  Deputy  for 
Software  Evaluation  (DSE)  establish  an  evaluator  team  consisting  of  at 


least  five  evaluators  knowledgeable  in  software  maintenance.  The 
program/modul e  hierarchy  is  established,  and  a  set  of  representative 
modules  is  selected  for  each  program  to  be  evaluated.  The  schedule 
for  the  evaluation  is  also  established  at  this  time.  The  DSE  briefs 
the  evaluator  team  on  the  procedures  and  assigns  the  necessary  iden¬ 
tification  information  for  this  specific  evaluation. 

The  function  of  the  calibration  phase  is  to  assure  a  reliable 
evaluation  by  assuring  that  each  evaluator  has  a  clear  understanding 
of  the  questions  on  each  questionnaire  and  their  specific  response 
guidelines.  Each  evaluator  completes  a  documentation  and  a  module 
source  listing  questionnaire  in  a  trial  or  calibration  evaluation. 
The  completed  questionnaires  are  reviewed  to  detect  areas  of  misun¬ 
derstanding  and  the  evaluation  teams  are  debriefed  on  the  problem 
areas. 

In  the  assessment  phase,  the  evaluation  teams  update  their 
calibration  test  questionnaires  based  on  the  results  of  the  cali¬ 
bration  debriefing.  The  teams  then  complete  the  remainder  of  their 
assigned  documentation  and  module  source  listing  questionnaires. 

In  the  analysis  phase,  the  DSE  accomplishes  the  conversion  and 
initial  data  processing  of  the  questionnaire  data.  The  statistical 
summaries  are  then  returned  to  the  test  director  for  detailed  eval¬ 
uation  and  preparation  of  the  final  report. 

Data  Collection  Procedures  (Questionnaires) 


The  questionnaires  used  for  assessing  the  software  documentation 
and  source  listings  require  rating  responses  following  the  rating 
scale  shown  below: 


A. 

Completely  Agree  (absolutely  no 

doubt) 

B. 

Strongly  Agree 

C. 

Generally  Agree 

D. 

Generally  Disagree 

E. 

Strongly  Disagree 

F. 

Completely  Disagree  (absolutely 

no  doubt) 

In  addition  to  a  rating  response,  the  individual  evaluators  may  elect 
to  submit  a  written  comment. 

Software  documentation  questionnaire.  This  questionnaire  is  used 
to  evaluate  the  overall  format  and  the  content  of  the  documentation 


(not  including  source  listings)  for  t^e  computer  program  being  eval¬ 
uated.  Although  the  information  reouired  to  answer  the  Software 
Documentation  Questionnaire  may  be  spread  out  among  several  distinct 
documents,  the  primary  information  sources  which  are  always  considered 
a  part  of  the  evaluation  are  the  program  functional /detail  ed  design 
specifications  and  the  program  maintenance/operational  procedures. 
Contractor  programming  conventions  should  also  be  made  available.  The 
documentation  which  is  to  be  evaluated  should  be  specified  to  the 
evaluator  by  the  DSE  prior  to  the  calibration  test.  Appendix  F  con¬ 
tains  the  list  of  statements  in  this  questionnaire. 

Module  source  listing  questionnaire.  This  questionnaire  is  used 
to  evaluate  the  overall  format  and  content  of  the  source  listing  for 
the  program  module  being  evaluated,  and  to  evaluate  the  consistency 
between  the  module's  documentation  and  source  listing.  The  program 
modules  which  are  to  be  eval"ated  are  specified  to  the  evaluator  prior 
to  the  calibration  test.  Appendix  F  contains  the  list  of  statements 
in  this  questionnaire. 

Formatting  of  results.  Once  all  data  are  gathered,  they  are 
weighted  as  specifi'e3  in  the  AFTEC  handbook,  and  average  values  are 
calculated.  At  this  point  the  data  should  be  formatted  for  easy 
interpretation  of  results.  A  particularly  effective  means  for  this  is 
with  bar  graphs.  Figure  3-2  shows  an  example  of  source  listing 
results  (as  excerpted  from  the  SAC  air  refueling  part- task  trainer 
lOT&E).^ 


SOFTWARE  EVALUATION:  TEST  DIRECTOR  CONCERNS 
Phase  of  Test  Considerations 

One  of  the  most  important  phases  of  test  considerations  relative 
to  software  is  configuration  management.  This  is  because  a  large  por¬ 
tion  of  software  evaluation  requires  an  accurate  correlation  between 
descriptive  information  (documentation/source  listings)  and  the 
program,  as  it  exists  functionally,  in  order  to  facilitate  post- 
delivery  life-cycle  support  by  software  support  personnel.  Detailed 
requirements  for  software  configuration  management  are  contained  in 
MIL-STD-1644(TD) ,  "Military  Standard  for  Trainer  System  Software 
Development." 


Lambert,  A.  G. ,  Jr.,  Amisano,  R.  P. ,  Burch,  N.  T. ,  i  Zimick,  D. 
C.  Initial  operational  test  and  evaluation  B-52  air  refueling  part 
task  trainer  (SAC  Project  77-SAC-333  ).  Castle  AFB,  CA:  4200"  TES, 


SOURCE  LISTING 


MODULARITY 
(Avg.  4.34) 


DESCRIPTIVENESS 
(Avg.  3.52) 


CONSISTENCY 
(Avg.  4.19) 


SIMPLICITY 
(Avg.  3.38) 


EXPANDABILITY 
(Avg.  4.03) 


Data/Control  -  3.93- 
Processing  -  4.50  — 


Preface  Block  -  2.98 - 

Inbedded  Comments  -  3.70 
Implementation  -  4.41 - 


External  -  3.96- 
Internal  -  4.49 


General  Coding  -  3.36— 
Singular  Coding  -  4.48 
Size  -  1.93 - 


General  -  4.36 - 

Processing  -  3.77- 


INSTRUMENTATION  Processing  -  3.84 

(Avg.  3.79)  Control  -  3.74 - 


Figure  3-2.  Example  format  for  source  listing  results. 


A  related  concern  has  to  do  with  the  high  probability  that  the 
software  as  first  implemented  will  need  to  be  changed  and  updated  to 
reflect  changes  in  aircraft  parameters,  tactics,  doctrine,  and  any 
other  areas  which  impact  task  performance  in  the  device.  Often  these 
changes  require  expeditious  implementation,  thereby  requiring  that  the 
simulator  system  software  be  designed  to  facilitate  efficient  change 
over  its  life  cycle.  It  is  important  to  note  that  future  software 
modifications  will  have  to  be  implemented  by  personnel  not  associated 
with  the  original  development  effort. 


There  are  two  planning  documents  of  particular  importance  ►. ith 
regard  to  software  test  and  evaluation.  These  documents,  which  should 
be  available  to  the  test  director,  are  the  CRISP  (Computer  Resou'tes 
Integrated  Support  Plan)  and  the  0/S  CMP  (Operational /Support  Config¬ 
uration  Management  Procedures).  These  documents  are  intended  to 
define  what  will  be  needed  downstream  to  support  the  computer  system 
and  to  maintain  accurate  configuration  management  for  the  system. 
AFLC  regulation  800-21,  "Management  and  Support  Procedure  for  Computer 
Resources  Used  in  Defense  Systems,"  also  contains  useful  information 
which  may  help  to  define  terms  and  guide  the  software  evaluator  to 
additional  sources  of  information. 


Personnel  Requirements 

As  is  the  case  with  many  other  areas  of  evaluation,  personnel 
requirements  will  vary  depending  on  system  complexity.  This  applies 
even  more  so  to  the  software  area.  One  way  to  determine  personnel 
requirements  is  to  select  a  good  DSE  (Deputy  for  Software  Evaluation) 
and  give  that  individual  the  responsibility  to  define  what  is  needed. 
Certain  of  the  characteristics  and  considerations  of  a  good  DSE  have 
been  identified  in  the  AFTEC  handbook.  These  are  excerpted  below: 

(a)  The  deputy  for  software  evaluation  (DSE)  should  be  brought 
on  board  early  to  assist  in  detailed  software  OT&E  planning 
and  to  become  familiar  with  the  system. 

( b)  It  is  imperative  that  the  software  test  manager  and  the  DSE 
have  a  good  working  relationship  with  each  other,  the 
contractor,  and  the  program  manager. 

(c)  It  is  imperative  that  the  DSE  is  a  self-motivator,  .f  not, 
test  team  motivation  becomes  a  problem. 

( d)  The  DSE  must  be  dedicated  to  the  test  for  the  entire  test 
period  including  final  report  writing. 

(e)  The  DSE  should  be  an  AFTEC/MAJCOM  resource  of  equal  rank  to 
the  deputy  for  logistics  and  the  deputy  for  operations. 


B.  SOF^ARE  USABILITY 


SOFTWARE  USABILITY  EVALUATION  ELEMENTS 

Software  usability  is  defined  as  the  extent  to  which  software 
designated  to  perform  a  support  function  is  effective  in  performing 
that  function  and  is  "usable"  by  the  Air  Force  operator.  This  eval¬ 
uation  normally  concentrates  on  an  analysis  of  the  adequacy  and  effec¬ 
tiveness  of  nonmission  software  (e.g.,  off-line  diagnostics,  ATE 
software)  in  terms  of  operator-machine  interface  and  functional  per¬ 
formance.  These  two  areas  are  discussed  further  below. 

Software  Operator-Machine  Interface 


This  evaluation  element  considers  the  adequacy  of  that  part  of 
software  design/implementation  which  affects  interaction  between  a 
computer-driven  system  and  its  operator.  It  is  divided  into  six  sub¬ 
elements  of  evaluation  concern  which  address  various  areas.  Each  of 
these  subelements  is  defined  and  discussed  in  the  subsequent  eval¬ 
uation  methods  section. 

Software  Functional  Performance 


As  a  usability  concern,  software  functional  performance  refers  to 
its  capability  to  carry  out  its  intended  purposes.  At  present,  this 
area  is  not  well  defined  in  AFTEC's  software  evaluation  handbook. 
However,  the  test  director  should  consult  AFTEC  softv/are  specialists 
to  determine  the  current  status  of  developments  relative  to  functional 
performance  evaluation. 


SOFTWARE  USABILITY  EVALUATION  METHODS 

The  methodology  for  evaluating  the  software  portions  of  the 
operator-machine  interface  ’>  based  on  the  use  of  a  closed  form 
questionnai re  with  optional  written  comments.  This  questionnaire  is 
designed  to  determine  the  extent  of  the  presence  of  certain  desirable 
attributes  in  a  given  system.  Appendix  G  contains  a  listing  of  these 
questionnaire  statements. 

The  desirable  attributes  addressed  by  the  questionnaire  are 
divided  into  six  subelements:  assurabil ity ,  controllability,  workload 
reasonabil ity ,  descriptiveness,  consistency,  and  simplicity.  A 
complete  understanding  of  the  definitions  of  these  subeiements  is  of 
prime  importance  to  an  accurate  evaluation;  thus,  the  evaluator  should 
study  these  definitions  carefully. 
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Assurability 


A  computerized  system  contains  the  quality  of  assurability  to 
the  extent  that  it  aids  the  operator  in  validating  data,  avoiding 
errors,  and  correcting  errors  once  made.  A  system  which  has  been 
designed  to  aid  the  operator  in  error  avoidance  may  or  may  not  have 
good  assurability.  A  system  should  also  be  designed  so  that  errors 
are  easy  to  correct  and,  above  all,  so  that  errors  do  not  have 
catastrophic  effects. 

Controllability 


A  computerized  system  contains  the  quality  of  controllability  to 
the  extent  that  it  allows  the  operator  to  direct  the  operations  of  the 
machine.  The  operator  must  be  able  to  direct  or  control  the  operation 
of  the  machine  in  order  to  utilize  it  effectively  and  efficiently. 

Workload  Reasonability 


A  computerized  system  contains  the  quality  of  workload  reasona¬ 
bility  to  the  extent  that  the  tasks  required  of  the  operator  are 
within  the  operator's  capability  and  require  the  operator  to  perform  a 
useful,  meaningful  role.  Optimum  design  of  a  system  which  involves  an 
operator  and  a  computerized  machine  takes  advantage  of  the  best  capa¬ 
bilities  of  both;  the  machine  to  perform  repetitive  tasks  rapidly, 
and  the  operator  to  make  command  decisions  involving  unusual 
si tuations. 

Descriptiveness 

A  computerized  system  contains  the  quality  of  descriptiveness  to 
the  extent  that  the  operator  has  available  adequate  explanations  of 
every  function  the  operator  is  required  to  perform  and  every  function 
the  machine  performs.  The  operator  need  not  be  informed  in  detail  of 
every  task  the  machine  performs,  but  there  are  certain  things  the 
operator  must  know  to  fulfill  the  mission.  The  questionnaire  relies 
upon  the  knowledge  of  the  operator  to  define  what  it  is  the  operator 
needs  to  know. 

Con si stency 

A  computerized  system  contains  the  quality  of  consistency  to  the 
extent  that  the  behavior  of  the  machine  and  documentation  correspond 
to  the  expectations  of  the  operator.  There  should  be  a  near  one-to- 
one  correspondence  between  what  the  machine  does,  what  the  documen¬ 
tation  says  it  will  do,  and  what  the  operator  has  been  trained  to 
expect  the  machine  to  do.  Furthermore,  the  documentation  normally 
available  to  the  operator  should  agree  with  that  which  the  operator 
has  been  trained  to  expect. 


Si'mpl  1C1  ty 


A  computerised  system  contains  the  quality  of  simplicity  to  the 
extent  that  the  information  presented  to  the  operator  or  entered  by 
the  operator  is  grouped  into  short,  readily  understandable  structures. 
Complicated  data  structures,  data  entry  formats,  or  operator  manuals 
all  require  the  operator  to  spend  more  time  in  developing  an  under¬ 
standing  of  the  system,  and  may  have  a  tendency  to  confuse  the  opera¬ 
tor  as  well . 

The  above  identified  subelements  have  been  grouped  logically  by 
AFTEC  into  factors  of  "operability"  and  "communicativeness."  A  com¬ 
puterized  system  contains  the  quality  of  operability  to  the  extent 
that  the  operator  is  in  control  of  the  operator-machine  interface. 
Operability  is  the  sum  of  assurabil ity ,  controllability,  and  workload 
reasonability.  A  computerized  system  contains  the  quality  of  com¬ 
municativeness  to  the  extent  that  the  transfer  of  information  between 
the  operator  and  the  machine  is  concise  and  complete.  Communica¬ 
tiveness  is  the  sum  of  descriptiveness,  consistency,  and  simplicity. 
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BITE  RATE  CODES 


The  following  is  a  listing  and  definition  of  BITE  Rate  Codes  used 
when  collecting  BITE  Reliability  Data. 

Code  Definition 

B1  BITE  indicated  a  problem. 

B200  BITE  should  have,  but  did  not,  indicate  a  problem. 

(If  code  B1  is  used,  a  third  code  character  is  required,  as  follows.) 
B13  BITE  isolated  the  problem  to  the  required  level. 

B14  BITE  did  not  isolate  the  problem  to  the  required  level. 

(If  codes  B13  or  B14  are  used,  a  fourth  code  character  is  required.) 

( For  code  B13; ) 

B135  BITE-indicated  problem  is  confirmed. 

B136  BITE-indicated  probl  em  is  not  confi  rmed ,  i  .e . ,  the  "  faul  ty" 

component  was  not,  in  fact,  faulty  (CND),  but  another 
component  was  faulty. 

B137  BITE-indicated  problem  is  not  confirmed  (CND),  and  there 

was  no  malfunction  at  all. 

(  For  code  B14: ) 

B14ft  BITE  did  not  isolate  the  problem  to  the  required  level, 

but  there  was,  in  fact,  a  confirmed  problem. 

B149  BITE  indicated  a  problem,  but  there  was  no  problem. 

Note:  The  third  or  fourth  code  characters  may  not  be  available 
until  after  the  corrective  action  taken  information  is 
available  from  the  contractor. 
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MTBM  FORMULA 


The  six  versions  of  the  MTBM  described  in  Chapter  2(A)  are  all 
calculated  using  the  same  basic  fonnula: 


MTBMA  = 


_ Operating  time _ 

Quantity  of  on-equipment  maintenance  occurrences 


where: 


Operating  time  =  system  elapsed  time  indicated  (ETI),  and  quan¬ 
tity  of  occurrences  =  the  total  number  of  maintenance 
occurrences  during  the  measured  interval . 


MTBCF  FORMULA 

The  mean  time  between  critical  failures  (MTBCF)  is  an  index  of 
the  operational  mission  reliability  of  the  ATD.  MTBCF  is  the  total 
operating  time  during  the  evaluation  divided  by  the  total  number  of 
critical  failures  during  that  time  and  is  calculated  as  follows: 

^  _ Operating  time _ 

Quantity  of  critical  failures 


where: 


Operating  time  =  system  elapsed  time  indicated  (ETI),  and  quan¬ 
tity  of  critical  failures  =  the  total  number  of  occurrences 
which  disrupt  the  completion  of  mission  objectives. 


Failures  in  redundant  components  are  included  in  MTBM  calcu¬ 
lations,  but  are  not  critical  failures  so  they  would  not  enter  into 
MTBCF  calculations.  Failures  of  equipment  due  to  improper  maintenance 
are  considered  occurrences,  but  are  not  critical  failures.  Secondary 
failures  also  are  not  considered  critical,  but  are  considered  occur¬ 
rences  and  will  be  included  in  MTBM  calculations.  Secondary  failures 
are  failures  that  occur  as  a  result  of  a  failure  in  some  other  element 
or  component.  For  example,  a  failure  in  the  voltage  regulation  cir¬ 
cuit  of  the  power  supply  may  damage  or  otherwise  cause  failures  in  the 
components  it  supplies.  Failures  due  to  improper  installation  are 
considered  occurrences,  but  are  not  critical  failures. 


BITE  Rate  Formul as 


The  number  of  occurrences  resulting  in  the  different  codes  are 
used  as  inputs  to  the  following  BITE  rate  formulas. 


BAIP 


_ (B135)  X  100 _ 

( B135+B136+B148+B200+B137+B149) 


BAOP 


(B135+B136-(-B148)  x  100 
(  B135+B136+B148+B200+B137+B149) 


BFDP 


(B200)  X  100 
(B135+B136+B148^-B200) 


BFA 


(B137-^B149)  X  100 
( B135+B136+B137+B148+B149) 
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MAINTAINABILITY  COMPUTATIONAL  FORMULAS 


Mean  Time  to  Repair  (MTTR) 


Total  corrective  maintenance 
clockhours  during  test  period _ 

Total  number  of  corrective  maintenance 
actions  during  the  test  period 


where  the  following  time  guidelines  normally  apply: 

1.  Time  spent  reading  TOs,  etc.,  is  included  if  directly  related 
to  the  maintenance  task.  Time  required  to  find  the  TO  is 
typically  not  included. 

2.  Time  spent  accumulating  tools  necessary  for  the  task  is 
included  if  they  are  available  in  the  immediate  area. 

3.  Time  spent  in  preparing  the  simulator  in  any  way  incidental 
to  the  task  ‘s  included. 

4.  Time  spent  in  direct  support  of  development  tasks,  e.g., 
repair  of  test  instrumentation,  is  not  included. 

5.  If  personnel  are  required  on  an  intermittent  or  a  sequenced 
basis,  the  time  assessed  for  the  task  includes  the  required 
standby  time  only  if  the  standby  time  is  of  a  type  or  dura¬ 
tion  which  prevents  these  personnel  from  performing  other 
productive  tasks. 

6.  If  an  item  is  damaged  or  maintenance  errors  are  induced  by 
design  complexity  or  improper  procedures,  the  time  will  be 
chargeable.  When  action  concerning  any  of  the  deficiencies 
has  been  completed,  the  time  will  not  be  deleted.  However, 
the  maintainability  prediction  will  incorporate  the  results 
of  any  subsequent  engineering  changes  that  would  affect  such 
times . 

7.  Corrective  maintenance  actions  will  include  all  those  actions 
documented  to  repair  inherent,  induced,  and  no-defect 
occurrences  (as  defined  in  reliability). 

8.  MTTR  excludes  delays  due  to  supply,  administration,  personnel 
nonavail abil ity,  and  transportation,  except  as  provided  pre¬ 
viously  within  this  section. 


MTTR  (critical) 


Corrective  maintenance  manhours 
expended  on  critical  failures 
Number  of  critical  failures 


r 


MTTR  (critical)  =  - 


Mean  Manhours  to  Repair  (MMTR) 


Total  corrective  maintenance 
manhours  during  test  period _ 

Total  number  of  corrective  maintenance 
actions  during  the  test  period 


where  time  guidelines  specified  for  MTTR  apply. 


Maintenance  Manhours  per  Training  Hour  (MMH/TH) 


MMH/TH  '  Total  maintenance  manhours 
Training  hours 

where  time  guidelines  for  MTTR  apply  as  well  as  the  following: 

1.  Manhour  expenditures  include  all  those  manhours  expended 

under  the  guidelines  described  under  MTTR. 

2.  Manhour  expenditures  are  only  counted  during  times  that  the 

simulator  is  scheduled  for  full  operational  testing,  i.e., 

during  the  reliability  demonstration  and  operational  effec¬ 
tiveness  testing. 

3.  Training  hours  are  counted  only  for  those  times  that  the 

simulator  was  used  in  a  full  operational  condition,  i.e.,  the 

reliability  denonstration  and  operational  effectiveness 
testing. 

4.  All  manhours  expended  from  beginning  of  test  (reliability 

demonstration  or  operational  effectiveness  tests)  until  the 
end  of  test  are  counced  if  they  fall  under  the  criteria 

described  under  MTTR.  End  of  test  will  be  when  all  main¬ 
tenance  actions  resulting  from  occurrences  during  test  are 

compl eted. 
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Maintenance  Manhours  per  Operating  Hour  (MMH/OH) 


MMH/OH  is  computed  for  the  six  categories  of  maintenance  action 
(preventive,  inherent,  induced,  no  defect,  all  failures,  and  total 
corrective)  using  the  following  formula: 


MMH/OH 


Direct  maintenance  manhours  (category) 
Operating  hours 


ADDITIONAL  MAINTAINABILITY  CONSIDERATIONS 
When  to  Start  Timing 


The  process  of  initiating  maintenance-team  action  after  a  fault 
is  inserted  has  been  described.  However,  that  description  did  not 
indicate  when  to  start  timing  the  maintenance  action.  This  is  some¬ 
times  complicated  by  procedures  involving  the  use  of  computer  checkout 
programs.  In  some  instances  these  programs  take  five  minutes  or  more 
before  an  answer  as  to  equipment  status  is  indicated.  Is  this  time  to 
be  counted  as  a  portion  of  the  restore  time,  or  merely  as  operational 
monitoring  time?  The  question  becomes  crucial  if  the  MTTR  requirement 
is  very  short,  e.g.,  15  minutes,  and  the  BITE  or  automatic  checkout 
time  is  the  greater  part  of  this  time.  The  question  must  be  resolved 
in  the  initial  test  planning  phase. 


QUALITATIVE  MAINTAINABILITY  CHECKLIST 

1.  Are  major  1  ine-'"epl acement  units  (LRUs)  located  to  facilitate 
total  system  inspection/checkout/troubleshooting? 

Z.  Does  system  design/instal 1 ation  contribute  to  ease  of  main¬ 
tenance  in  terms  of  location,  accessibility,  etc..  Consider: 

a.  Size  and  access  panel  s/doors ,  number  of  and  type  oT 
fasteners. 

b.  Size  and  weight  of  components,  adequacy  of  handles  or 
handholds,  required  span  of  reach,  height  above  or 
distance  from  work  surface. 

c.  Location  of  test  or  servicing  points  in  relation  to  work 
surface  for  test  or  servicing  equipment. 


d.  Adequacy  of  space  for  necessary  support  equipment. 


3.  Are  test  or  servicing  points  clearly  marked  to  reduce  chance 
of  induced  error? 

4.  Are  connectors  of  different  size,  keyed,  or  clearly  marked  to 
eliminate  swapping? 

5.  Are  connectors  visible  and  readily  accessible  to  reduce 
chance  of  cross-threading ,  etc.? 

6.  Are  there  hazards  in  terms  of  blind  spots,  shirp  edges, 
exposed  electrical  connectors/circuitry? 

7.  Can  the  system  be  checked  out  (operational  check,  trouble¬ 
shoot,  etc.)  by  not  more  than  two  technicians? 

8.  Are  support  equi pment/BITE  cues  and  indications  easily  read 
and  understood? 

9.  Are  environmental  conditions  such  as  noxious  fumes,  high 
noise  levels,  extreme  temperature,  etc.,  tolerable? 

10.  Describe  other  features  or  requirements  not  listed  above 
which  adversely  affect  system  maintenance: 


11.  Was  the  equipment  under  evaluation  considered  commercial  off- 
the-shel  equipment? 

12.  Remarks  (explain  questions  1-10  that  were  answered  nega- 
ti  vely) : 
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AVAILABILITY  COMPUTATIONAL  FORMULAS 
Inherent  Availability 

The  foil  owing  formula  is  used  to  calculate  A-j  for  the  device: 

^  _  _ MTBCF _ 

’  '  MTBCF+MHR  (critical) 


Scheduling  Availability 


Number  of  missions  scheduled  - 
number  of  missions  lost  + 
number  of  missions  added 

Number  of  missions  scheduled  + 
number  of  missions  added 

Mission  hours  scheduled  - 

_ hours  lost  hours  added _ 

Mission  hours  scheduled  +  hours  added 


Mission  Capable  Rate  Chart 


Figure  C-1  shows  a  sample  mission  capable  rate  data  collection 
chart.  Guidelines  for  completing  the  chart  are  as  follows: 


1.  The  first  entry  on  the  first  chart  is  the  time  and  status  of  the 
simulator  at  initiation  of  the  test  period, 

2.  Whenever  the  mission  capable  status  changes,  enter  in  the 
following  line  the  time,  new  status,  primary  contributing  work- 
unit  code  (WUC),  a  brief  description  of  the  cause,  and  initial  the 
entry. 

3.  Fnter  all  changes  of  simulator  status,  including  an  N/P  status  for 
non-possossed  time  (if  applicable). 

4.  Use  one  chart  per  day,  with  ?400  hours  as  Lne  daily  iinsi.u.i 
time.  Date  each  chart.  The  last  entry  of  the  day  should  be  ?400 
hours  with  a  N/C  status  entry  (no-change). 


5.  After  the  last  entry  of  the  day,  total  the  times  for  each  status 
and  complete  the  "totals  for  the  day"  section. 


MISSION  CAPABLE  CHART 


Date 


TIME  STATUS  WUC  COMMENTS  INITIALS 


TOTALS  FOR  THE  DAY 


A. 

Possessed  Time 

F. 

Flyable  Time  (C+D+E) 

B. 

Non-possessed  Time 

G. 

(NMCM)  Sch  Time 

C. 

FMC  Time 

H. 

MMCM  IJnsch  Time 

D. 

(PMCS)  Time  _ 

I. 

(NMCS)  Time 

E. 

(PMCM)  Time 

J  . 

NMC  Time  (G+Hrl ) 

Figure  C-! .  Sample  mi';';inn  capable  rate  data  collection  form. 


Mission  Capable  Rate  Formulas 


The  following  data  are  required  to  make  MCR  determinations: 

•  Possessed  clockhours. 

•  Status  clockhours. 

•  Descriptions,  WUCs,  and  job  control  numbers  of  primary  causes 
of  status. 

Using  the  data  on  clockhours  per  type  of  status,  and  the 
possessed  clockhours  data,  mission  capable  rates  are  calculated  using 
the  following  formulas: 


FMC 


FMC  clockhours 
Possessed  clockhours 


X  100 


iiMf-M  u  j  j  NMCM  scheduled  clockhours  ^ 

MMCM  scheduled  rate  =  -  X  100 

Possessed  clockhours 


NMCM  unscheduled  rate 


HMCM  unscheduled  clockhours 
Possessed  clockhours 


X  100 


NMCS  rate  = 


NMCS  clockhours 


Possessed  clockhours 


X  100 


PMCM  rate 


PMCM  clockhours 


Possessed  clockhours 


X  100 


PMCS  rate 


PMCS  clockhours_ 
Possessed  clockhours 


X  100 


(PfCM  +  PMCS  FMC)  clockhour^ 
Possessed  clockhours 


Flyable  rate 


X  100 
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MANPOWER  CALCULATION  PROCEDURE 


1.  Multiply  the  number  of  hours  in  each  shift  by  the  number  of 
personnel  required  on  that  shift  to  get  manhours  per  shift 
(each  shift's  minimum  manpower  situation  will  be  used). 

2.  Multiply  manhours  per  shift  by  days  per  month  (of  that  shift) 
to  get  monthly  manhours  for  each  situation. 

3.  Total  the  monthly  manhours  for  all  situations  to  determine 
total  minimum  manhours  per  month. 

4.  Divide  by  the  appropriate  availability  factors  to  determine 
minimum  manpower. 


SUPPORT  EQUIPMENT  EVALUATION  CHECKLIST 

1.  Is  the  item  easily  operated? 

2.  Was  the  proposed  SE  adequate  for  the  task  in  the  following  areas: 

a.  Depth  of  test  or  diagnostic  capability  versus  test  required 

to  ensure  proper  system  operation? 

b.  Range  of  SE  inputs  versus  system  range  of  operation? 

c.  SE  performance  parameters  (power,  accuracy,  precision)  versus 
system  performance  parameters? 

3.  Are  instructions  complete  and  adequate  for  SE  hookup  operation 

and  diagnosis? 

4.  Does  the  SE  (to  include  diagnostic  and  BITE  routines)  test  the 

system  (or  subsystem)  to  the  same  parameters  (voltage,  frequency, 
etc.)  as  those  to  which  the  system  is  supposed  to  operate? 

5.  Were  indications  or  cues  easily  understood? 

6.  Were  diagnostic  and/or  RITE  routines  easily  initiated? 

/.  Does  this  item  appear  to  be  corrosion  free? 

8.  Can  the  proposed  SE  "stand  alone"  and  not  be  supported  by  other 
common  or  special  units  (voltmeters,  freqtjoncy  counters)  while 
being  used? 

9.  Does  this  item  require  calibration? 


10.  Does  this  item  use  software  (computer  programming)? 

11.  Arc  there  similar  maintenance  or  service  functions  perfonun  ly 
other  SE? 

1?.  Is  there  an  alternate  method  which  would  not  require  use  ut  this 
equi pment? 

13.  Was  a  maintenance  task  performed  using  this  equipment? 

14.  Is  there  a  possible  safety  hazard  at  any  time  during  *rnnsi>or- 
tation  or  positioning  and  using  this  equipment? 

( Afiy  answer  of  "no“  to  any  of  the  questions  from  1  to  B,  and  any 
>nswer  or  "yes"  to  any  of  the  questions  from  9  to  14  requires  explana¬ 
tion  on  hack . ) 

'^^'CHNICAh  DATA  CONTENT  CHECKLIST 
Sat'^Unsat  (S/U) 

1.  Manuals  identify  aJJ  units  &  assemblies  by  location  &  function. 

2.  Mdiujal  s  provide  schematics  &  wi’^inq  diaorams  at  least  to  the 

LRU.  — —  ^  --  - 

3.  -tinual  s  describe  all  uncommon  oarts,  tools,  codes,  or  test 
uni ts.  . . 

_4.  Manuals  tell  how  to  detect,  localize,  iSL)late,  correc . 

c  hnckout . 

S.  '’anual  s  explain  what  to  chc^cj^,  what^to  expect  ^  how  to  correct. 

h.  'hinual  $  tell  what  may  go  wrong,  how  to  i'cevent  A  how  to 

rf‘(  over  . 

/.  '*;nuals  accur^^tely  list  men.  t(>ols,  matiriel  u^ed  in  eacLi  task. 
h.  ‘-'nnaals  ’ayout  cues/aids  effective  (’ffi  ir^rit  tra  uhl  nchoot  i  nq  . 

,  *’ar;ual  s  clruu'ly  descrihr  access,  breakdown  ^  .isseir.hly  lU'thods. 

ic.  /•  i  1  rof-rM,  y  conditic^ns,  pruc<’dur'f  s  ^  es<  r.i.t-s  /e*  ss 
’.1.  /■■ !  1  adjust,  ilion,  calit)*"ate  A  cT'-ckru*  pri  '  ’  Onwn . 


1?,  ♦'  -  ;'v*'  ‘  »  :iri usual  cl  1natos/condi lions . 

13.  t'^oasur^  ♦  Sta  :»'r  :u  ♦  s  '  L*qu  i  pnon  t  to  be  used, 

14.  Data  dw  Kqfvully  c- r/Mr? ) jod ,  qur.Kly  found,  &  readily  used. 

15.  Manual  ter''>s,  syrd^ol  s  are  cons^  .^.^nt  with  maintenance  data 
system . 

16.  Ml  procc  lures  are  consistent  with  expected  use  &  fail  ure 
rates . 

17.  All  procedures  are  consistent  with  planned  manning  &  workloads. 

18.  All  procedures  reflect  supply,  handling,  &  storage  practices. 

19.  Maintenance  block  diagrams  are  provided  for  each  equipment 
i  t  em . 

20.  Diagrams  describe  interconnections  A _ re1 ationships  between 

items. 

21.  Diagrams  identify  input-output  connections  between  subass'^m- 

blies.  ^ 

22.  Diagrams  give  designations  for  terminals,  jacks  ^  test  points. 

23.  Diagrams  show  voltage,  current,  K  waveform  at  jte^  j>oi  nt . 

24.  Diagrams  reflect,  are  compatible  with,  diagnostic  techniqu("s , 

25.  All  materials  are  consistent  with  svstem  maintenance  con.tpts. 


TKCHMICAL  DATA  STYLF  CHFCKLIST 
Sat/llnsat  (S/ll) 

1.  Manual  s  are  [danned,  designed,  1i  str ilMjt‘'d  for  easy,  dailv  u^e. 

2.  Print  is  jm  ■;  material  durable*;  nc'  transiiu  r^n  t/gl  ossy  par^-r. 

3.  Where  possible,  pocket  manuals  contain  s[m‘r  i  al  i  st  1  f  i  c 

data .  ^  — 

4.  Major  portions  of  the  manual  im  tabt^ed  and/or  subject  ii 

5.  Roth  detailed  table  of  contents  K  subject  index  are  prrivi  led. 


J6.  Indexes  are  symptom  oriented,  to  lead  from  trouble  to  solution. 

_7.  Instructions  are  in  step-by-step  rather  than  narrative  format. 

_8.  Each  procedure  in  the  manual  has  been  tried,  validated. 

_9.  Maintenance  procedures  avoid  unnecessary  testing  or  handling. 

10.  Instructions  balance  workload  among  personnel,  between  hands. 

11.  Instructions  fix  action  location  before  describing  the  action. 

12.  Tools,  testers,  &  material  are  listed  at  top  of  each  instruc- 
ti  on . 

13.  Dial,  meter,  swi tch  setti ngs  are  given  wherever  appropriate. 

M.  Warnings  A  cautions  are  given  in  the  sequence  encountered, 

15.  Feedback  loops  load  to  discovery/correction  of  probable  errors. 

16.  Technical  data  are  clear,  unambiguous,  require  no  inter- 
poTatiolru 

17.  All  language,  words,  K  symbols  are  short,  familiar,  &  concise, 

Ptjragraphs  are  with  frequent  run-in  &  side  htadinqs. 

19.  Titles,  subti ties,  &  headings  clearly  indicate  area  of 
coverage. 

?0.  Fold  type,  underlining,  and  spacing  for  salient  kev  words  ^ 
thoughts .  ’ 

?].  Tables,  charts,  ^  illustrations  are  used  wherever  practicable. 

2/'.  Photographs  show  unfamiliar  detail,  are-  rriO'uched  to  aid 
reader . 

?3  Drav/inqs  i  1  1  us  t  a  tr-  faniiliar  ite^'ns.  show  -^’ovo'^ent,  .  b  -  t  a  t  i  on  . 

2A.  r -pU  ded  views  show  part  l(U.itior,  d  i  sa y  opr. 

2‘  .  data  ar..  liT'- to- dat^' ,  ^ut  rrvisio'^  rr-ari'.  at  o  pro-- 

V i dod . 


FACILITIES  EVALUATION  CHECKLIST 


Sat/Unsat  (S/U) 

,  Facility  layout  miniinizes  mai ntenance/opoi^ati ons  interfef"onco . 

_2.  Layout  minimizes  place- to-place  movement  of  men  &  equipment, 

_3.  Layout  provides  adequate  bench  maintenance,  shop,  storage 
space. 

_4.  Layout  allows  visual  &  voice  contact  tu^tween  team  rnombers. 

_5.  Layout  allows  access  to  most  sides  of  all  items  of  equipment. 

6.  All  spacing  is  olanned  for  likely  clothing,  loads,  clutte»", 
_  etc. 

^7.  Stock  room/ too i  crib  locations  are  convenient  to  all  work  arenn. 

Special  storage  is  provided  for  L,arardous  or  contam  nable 
7  terns . 

__9,  Kick -space,  k'u-v  room,  wri  t1  no  surfaces ,  etc ,  are  adenuate. 

^10.  Passageways  adeuo.ite  for  carts,  ^^tands,  etc.  K  their  loa^s. 

_11.  Passages,  cc.rners  allow  ca-'cmete  rcmiOval  of  largrst 

i  tem. 

]?.  f'^assaces/door allow  easy  3cc^ss_  to  E.  escape  from  al  ■  - 

places. 

13.  L'orkspace  is  planned  primarily  for  rtand'cc  -itting  tasks. 

14.  Workspuice  is  •  O-quati-  dn^pite  ,  n;."-  irawrrs,  or 

15.  Worksparr^s  /o  ^Vre  f  hiacands,  Ts  I  r|j(  H  '^s  ^  ch^rp  ecLoe'''' .  -  t  . 

16.  ‘‘pat  c  <e:*  e'.*;'  i^'-r  renijired  frawlinn,  clird'if'-;, 

e  tc . 

17.  rhairs  are  ^  0-;  .d'f'r'r-  Mien  'd  'r  c^  t  (' f  the  shdft. 

I>^.  1 1 1  ui'm' na  t  i  (m-  Kp'O  .-dt  or  work  su^  O;  f'-  ,  :li  d  . 

\^K  Internal  Pvpd.t  '  me  H  nn  asrisi  '’Mirt<  fcr'  f  ,  q1  I'-n  is  avci 


« 


14.  Were  container",  .  i  i  r  renter  of  balance,  slinq-lift  facili¬ 

ties,  forklift  entry  points,  etc.,  when  required? 

if.  !s  the  supply  of  --j^erial  complete  in  terms  of  type  of 

ii’iaterial  and  adecjuaie  in  i"  ‘int? 

ir>.  Are  space  and  tools  to  allow  the  construction  ot 

crates  ar- :  containers? 

17.  Is  heavy  capacity  nuch  as  forklifts,  elevators,  a’-r 

hoists  sufficient  for  ; '''e’^'en ts? 


18. 

Are 

1  oad i nq  ' un 1 

rading  no.  ►  \ 

•:  ‘  U.iate? 

Are 
(  1  f 

special  p  f  iV  s i c  a 1  pa  c  •  - 
reon  i  ‘^rd )  ? 

er  handl  ing  precaut.ions  prpv  v  ec 

security  re 

q'l ’  rf^tpen r s  ^ 

V  for? 

?  ' 

Is 

onv  i  rr- fir  ten  to 

‘  pro  ti‘c  tie"' 

rVP-  , 

C  i  . 

Have  written  pr 

^H'ediires  prr 

'  r  'v  ^-r-d  for  on-  si  tc  use? 

L’3 . 

^0 

the  ;  •■(i-. 

^,rovide  fnr: 

a . 

Prope»^  1  r 

e  tr  . 

.  1  "1  ;  ,  t( ,  if,. 

"'■a  f''  (1 1  i  n  q  p r  v c  actions,  a d d  r e  s  -  e  s  , 

b. 

’  n  S’  :  V  n  sn- 

jritv  C  r*  '  ■ 

in.  y'? 

-  in  .V'dMic-na!  senu^'ity  areas,  or  jr^-c. ^ 

n  ..r  vMO^  .iUdr-ii  (-r  v  i  n/rntal  protrCtiorC^ 

P^v  o  r-ri>e(_‘e!jr<'s  beer  i; st  .i'- 1  i  shed  t('»'  ' '  1  i  t^i ry  /  connorL  1  a  1 

sn  i  ;'".n'n  1 1* 

•rf  t  c  r-n  s  i  1  T  V  r  itonin  \  i  ,  tii":e  charier  items,  items  with 

t  :  i  f  i  tmI  -  hrO  t  life,  t  - 1  n  .  i  *  ■  ■  i  t  i  ( -  d  ? 
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O&S  COST  EVALUATION  FORMULAS 


Simulator  Maintenance  Manpower  Costs 

Cost  1  =  (no.  of  airmen)  X  (airman  pay  factor) 

=  (no.  of  officers)  X  (officer  pay  factor) 

=  (no.  of  civilians)  X  (civilian  pay  factor) 


Cost  2 
Cost  3 
Cost  4 


(no.  of  CFS  [contractor  field  service]  and  FSR  [field 
service  representatives])  X  (sum  of  O&M  contract  costs 
per  man  year) 


Cost  Tot  =  Cost  1  +  Cost  2  +  Cost  3  +  Cost  4 


Replenishment  Spares 

Cost  =  X  OP  hrs/year  X  cond  rate  X  UC  UCi 

MTBM  (induced) 

where: 

•  is  the  unit  equipment  (number  of  simulators  per  wing  or 
squadron) . 

•  OP  hrs/year  is  the  scheduled  number  of  individual  annual  simu- 
1ator  operating  hours. 

•  Cond  rate  is  the  replenishment  spares  predicted  condemnation 
rate . 

•  U£  is  the  unit  cost  of  the  replenishment  spares. 

•  KTBM  (induced)  is  the  predicted  MTBM  of  the  replenish" 
spares. 

•  UCi  is  initial  spares  acquisition  cost  by  unit. 


Simulator  Maintenance  Materiel 


Cost  =  UE  X  OH  X  MPOH 
where; 

•  unit  equipment  (number  of  simulators  per  wing  or 
squadron) . 

•  M  ^  annual  operating  hours  per  simulator. 

•  MPOH  i  s  the  maintenance  materiel  expended  per  operating  hour. 


Support  Equipment 

Cost  =  UE  X  (SE  cost  factor) 

where: 

•  is  the  unit  equipment  (number  of  siiiuilitors  per  wing  or 
squadron) . 

t  ^  cost  factor  is  the  cost  of  requisition  and  replacement  of 
simul ator  SE. 

Simulator  Facility 
Cost  =  AREA  X  BFAC 
where : 

•  AREA  is  the  facility  floorspace  required  in  direct  support  of 
AID  operations  and  maintenance,  including  the  floorspace  con¬ 
sumed  by  the  trainer  itself.  This  variable  is  expressed  in 
square  feet. 

•  base  peculiar  O&M  planning  fact^^r  which  is  used  to 
account  for  geographical  differences  when  p»'()r;ra!nnn’ng  facili*- 
ties  maintenance  cost.  This  factor  is  expr^^^^sed  in  current 
year  dol 1 ars. 


I 


Electrical  Power 


Cost  =  PWR  X  PCOST  X  OPIIRS  X  UE 
whore: 

•  PWB  1 s  the  predicted  hourly  el ectrical  power  requi red 
for  operation  and  maintenance  of  a  simulator. 

•  PCPSJ  is  the  cost  of  the  above  electrical  power  per  unit, 

•  QPHRS  is  the  predicted  annual  operating  hours  of  a  simulator. 

•  yE  is  the  unit  equipment  (number  of  simulators  per  wing  or 
squadron)  . 
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SOFTWARE  DOCUMENTATION  QUESTIONNAIRE 


MODULARITY  QUESTIONS 

1.  The  documentation  includes  a  separate  part  for  the  description  of 
external  interfaces. 

2.  The  documentation  includes  a  separate  part  for  the  description  of 
each  major  function. 

3.  The  documentation  includes  a  separate  part  for  the  description  of 
the  program  global  data  base. 

4.  Major  parts  of  the  documentation  are  essentially  self-contained. 

5.  The  documentation  has  been  physically  separated  into  (sets  of) 
volumes  each  with  a  distinct  purpose. 

6.  The  documentation  indicates  that  each  global  data  structure  is 
partitioned  into  functionally  related  sets  of  variables. 

7.  The  documentation  indicates  that  storage  locations  are  not  used 
for  more  than  one  type  of  data  structure. 

8.  The  program  control  flow  is  organized  in  a  top  down  hierarchical 
tree  pattern. 

9.  The  documentation  indicates  that  program  initialization  pro¬ 
cessing  is  done  by  one  (set  of)  moduie(s)  designed  exclusively 
for  that  purpose. 

10.  The  documentation  indicates  that  program  termination  processing 
is  done  by  one  (set  of)  module(s)  designed  exclusively  for  that 
purpose . 

11.  The  documentation  indicates  that  program  I/O  is  done  by  one  (set 
of)  module(s)  designed  exclusively  for  that  purpose. 

12.  The  documentation  indicates  that  program  error  processing  is  done 
by  one  set  of  modules  designed  exclusively  for  that  purpose. 


11)1 


DESCRIPTIVENESS  QUESTIONS 

13.  Each  physically  separate  part  of  the  documentation  includes  a 

useful  table  of  contents. 

14.  Each  physically  separate  part  of  the  documentation  includes  a 

useful  glossary  of  major  terms  and  acronyms  unique  to  that  docu¬ 
ment. 

15.  Each  physically  separate  part  of  the  documentation  includes  a 

useful  index. 

16.  It  is  easy  to  locate  specific  information  within  the  documen¬ 
tation  . 

17.  The  documentation  includes  a  useful  version  description  document. 

18.  A  useful  master  list  is  available  which  identifies  all  software 

documentation . 

19.  Any  dynamic  allocation  of  resources  (storage,  timing,  priority, 
hardware  services,  etc.)  is  explained  in  the  documentation. 

20.  Timing  requirements  for  each  major  function  of  the  program  are 
explained  in  the  documentation. 

21.  Storage  requirements  for  each  major  function  of  the  program  are 
explained  in  the  documentation . 

22.  The  inputs  to  each  module  are  explained  in  the  documentation. 

23.  The  processing  done  by  each  module  is  explained  in  the  documen¬ 
tation  . 

24.  The  outputs  from  each  module  are  explained  in  the  documentation. 

25.  Special  processing  considerations  (error,  interrupt,  etc.)  of 
each  module  are  explained  in  the  documentation. 

26.  There  is  a  flow  chart  (or  equivalent)  for  each  module  which  ade¬ 
quately  illustrates  the  inputs,  general  processing,  and  outputs 
for  the  module. 

27.  Program  initialization  and  termination  processing  is  explained. 

28.  Recovery  from  externally  generated  error  conditions  which  could 
affect  the  program  is  explained. 


J 


2^^.  The  process  of  recovering  from  intemany  generated  error  con¬ 
ditions  is  explained. 

30.  Input  of  program  data  is  explained. 

31.  Output  of  program  data  is  explained. 

32.  There  is  a  useful  set  of  charts  which  show  the  general  program 
control  and  data  flow  hierarchy  among  all  modules. 

33.  There  is  a  master  list  (chart,  table,  section,  etc.)  identifying 
where  each  global  variable  is  used. 

34.  The  global  variable  master  list  includes  information  about  each 
global  variable  such  as  type,  range,  scaling,  units,  etc. 

35.  The  use  of  any  complex  mathematical  model  (technique,  algorithm) 
is  explained  in  the  documentation. 

36.  The  documentation  on  each  complex  mathematical  model  includes 
information  such  as  a  derivation,  accuracy  requirements,  stabil¬ 
ity  considerations  and  references. 


CONSISTENCY  QUESTIONS 

37.  It  appears  that  a  useful  set  of  standards  has  been  follov^ed  for 
the  development  of  the  documentation. 

38.  It  appears  that  a  set  of  standards  has  been  followed  for  the 

construction  of  all  (program  and  module)  flowcharts  (or 

equival ent) . 

39.  Documentation  of  each  major  functional  part  of  the  program 

follows  the  same  format. 

40.  The  format  of  the  documentation  reflects  the  organization  of  the 
program . 

41.  It  appears  that  programming  conventions  have  been  established  for 
the  interfacing  of  modules. 

42.  It  appears  that  programming  conventions  have  been  established  for 
I/O  processing. 

43.  It  appears  that  design  conventions  have  been  established  tor 

error  processing. 


44.  A  naming  convention  for  modules  appears  to  have  been  used. 

45.  A  naming  convention  for  global  variables  appears  to  have  been 
used. 


SIMPLICITY  QUESTIONS 

46.  The  terminology  used  in  the  documentation  to  describe  the  program 
is  easily  understood. 

47.  The  documentation  is  physically  organized  as  a  systematic 
description  of  the  program  from  levels  of  less  detail  to  levels 
of  more  detail  . 

48.  Each  part  (sentence,  paragraph,  subsection,  section,  chapter, 
volume,  etc.)  of  the  documentation  tends  to  express  one  central 
i  dea . 

49.  The  amount  of  cross  referencing  among  parts  of  the  documentation 
contributes  to  the  understandabil  i ty  of  the  program  description. 

50.  The  documentation  indicates  that  the  program  source  language  is  a 
high  order  language  (HOL). 

51.  The  documentation  indicates  that  the  use  of  recursi ve/ reentrant 
programming  techniques  is  not  excessive. 

52.  The  documentation  indicates  that  each  program  module  is  designed 
to  perform  only  one  major  function. 

53.  The  documentation  indicates  that  resource  (storage,  timing,  tape 
drives,  disks,  consoles,  etc.)  allocation  is  fixed  throughout 
program  execution. 

54.  The  documentation  indicates  that  the  control  flow  among  modules 
is  easy  to  follow. 

55.  The  timing  scheme  designed  for  the  program  is  easily  understood 
from  the  documentation. 

56.  The  program  is  designed  so  that  modules  are  not  interrupted 
during  execution. 

57.  It  is  evident  from  the  documentation  that  a  knowledge  of  mathema¬ 
tics  beyond  basic  algebra  is  not  required  to  understand  the 
mathematical  functions  performed  by  the  program. 
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EXPANDABILITY  QUESTIONS 


58.  A  numbering  scheme  has  been  adopted  which  allows  for  easy  addi¬ 
tion  or  deletion  of  narrative  parts  of  the  documentation. 

59.  Graphic  materials  (figures,  charts,  lists,  etc.)  are  physically 
separate  (e.g.,  on  separate  pages)  from  narrative  description. 

60.  A  numbering  scheme  has  been  adopted  which  allows  for  easy  addi¬ 
tion  or  deletion  of  graphic  materials. 

61.  The  program  timing  scheme  appears  to  be  flexible  enough  to  allow 
for  modifications  (e.g.,  reorganization,  addition,  deletion  of 
functional  parts) . 

62.  There  is  a  reasonable  time  margin  for  each  major  program  function 
(rate  group,  time  slice,  priority  level,  etc.). 

63.  Documentation  narrative  explains  the  procedures  for  altering 
basic  data  storage  sizes. 

64.  The  program  has  been  designed  to  allow  for  an  increase  i-!  storage 
utilized  before  storage  capability  is  exceeded. 

65.  Those  modules  dependent  upon  data  structure  sizes  are  identified. 

66.  The  program  has  been  designed  so  that  functional  parts  may  be 
easily  added  or  deleted. 

INSTRUMENTATION  QUESTIONS 

67.  There  is  a  separate  part  of  the  documentation  for  the  description 
of  a  program  test  plan. 

68.  There  is  a  separate  part  of  the  documentation  for  the  description 
of  sample  test  data. 

69.  There  is  a  separate  part  of  the  documentation  for  the  description 
of  program  support  tools  which  would  aid  in  testing  program. 

70.  A  set  of  test  procedures  to  be  used  for  program  checkout  is 
expl ained . 

71.  The  set  of  test  procedures  provides  useful  unit  testing  infor¬ 
mation  . 

72.  The  set  of  test  procedures  provides  useful  information  on 
1 imi tations/ incompl eteness . 


73.  The  program  has  been  designed  with  the  capability  to  display 

inputs  and  outputs  in  summary  form. 

74.  The  documentation  describes  a  standardized  set  of  program  U^st 
data  (input  and  ouput)  that  has  been  designed  to  exercise  the 
program . 

75.  The  documentation  indicates  that  the  program  has  been  designed  Xx) 

include  software  test  probes  to  aid  in  identifying  processing 

performance . 

76.  Error  checking  within  the  program  has  been  designed  to  include 

such  features  as  diagnostic  reporting,  I/O  parameter  checking, 
runtime  index  range  checking,  etc. 


GENERAL  QUESTIONS 

77.  Modularity  as  reflected  in  the  program  documentation  Lontrihu^es 
to  the  maintainability  of  the  program, 

78.  Descriptiveness  as  reflected  in  the  program  documentation  contri¬ 
butes  to  the  inaintainabil  ity  of  the  program. 

”^9.  Consistency  as  reflected  in  the  program  docu'iicn^a  1 1  on  contributes 
to  the  maintainabil ity  of  the  program. 

RO.  Simplicity  as  reflected  in  the  program  documorta ti on  contributes 
to  the  na i  n ta  1  nabi  1  i  ty  of  the  prograiri. 

81.  Expandabil i ty  as  reflected  in  the  program  documentation  contri¬ 
butes  to  the  maintainabil ity  of  the  program. 

8?.  Instrumentation  as  reflected  in  the  program  documentation  contri¬ 
butes  to  the  maintainability  of  the  program. 

83.  Overall,  it  appears  that  the  characteristics  of  the  program  docu¬ 
mentation  contribute  to  the  maintainability  of  the  program. 


SOFTVJARE  SOURCE  LISTING  QUESTIONNAIRE 


MODULARITY  QUESTIONS 

1.  Functionally  related  data  elements  have  been  organized  into  logi¬ 
cal  data  structures. 

2.  The  concepts  of  structured  programming  have  been  applied  to  the 
control  structures  in  this  module. 

3.  The  use  of  technioues  which  involve  the  sharing  of  memory  loca¬ 
tions  (e.g.,  overlay,  equivalence,  same  area)  is  not  excessive. 

4.  The  use  of  global  data  in  this  module  is  not  excessive. 

5.  The  number  of  entry  points  of  this  module  is  not  excessive. 

6.  The  number  of  exit  points  of  this  module  is  not  excessive. 

7.  This  module  performs  only  related  functional  tasks. 

8.  Each  functional  task  of  this  module  is  an  easily  recognizable 
block  of  code. 

9.  It  appears  that  each  iteration  block  within  this  module  has  a 

single  entry  point. 

10.  It  appears  that  each  iteration  block  within  this  module  has  a 

single  exit  point. 

11.  It  appears  that  each  decision  block  within  this  module  has  t. 

single  entry  point. 

12.  It  appears  that  each  decision  block  within  this  module  has  a 

single  exit  point. 

13.  When  this  module  completes  execution,  control  is  returned  to  the 
cal  1  ing  modul e . 

14.  The  use  of  the  same  variable  for  both  input  and  output  is  not 
excessive  in  this  module. 


DESCRIPTIVENESS  QUESTIONS 

16.  Inputs  to  this  module  are  described  in  a  preface  block. 

16.  Outputs  from  this  module  arc  described  in  a  preface  block. 
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17.  The  purpose  of  this  module  is  described  in  a  preface  block. 

18.  Modules  which  call  this  module  are  identified  in  a  preface  block. 

19.  Modules  which  are  called  by  this  module  are  identified  in  a  pre¬ 
face  block. 

20.  Limitations  (accuracy,  timing,  data  I/O,  etc.)  are  described  as 
appropriate  in  a  preface  block. 

21.  Any  special  processing  (e.g.,  multiple  entry/exit,  error 
handling,  algorithm  peculiarities,  etc.)  is  described  in  the  pre¬ 
face  block  and  is  understandable. 

22.  Documentation  information  (module  name,  programmer,  algorithm 
references,  revision  data,  etc.)  is  identified  as  appropriate  in 
a  preface  block. 

23.  The  comments  in  this  module  contain  useful  information. 

24.  The  quantity  of  comments  does  not  detract  from  the  legibility  of 
the  source  listings. 

25.  Transfers  of  control  and  destinations  are  clearly  explained. 

26.  Machine-dependencies  are  clearly  commented. 

27.  Imbedded  comments  describe  each  function  (block  of  code)  within 
thi s  modul e. 

28.  Attributes  of  each  variable  used  in  this  module  are  described  by 
comments  and/or  source  language  declarations. 

29.  Error  processing/exits  are  clearly  identified  and  explained. 

30.  It  appears  that  a  standard  for  module  organization  has  been 
followed  within  this  module. 

31.  Variables  are  declared  in  a  specification/declaration  section. 

32.  Variable  names  are  descriptive  of  their  functional  use. 

33.  The  module  code  is  indented  within  control  structures  to  show 
control  flow. 

34.  Statement  labels  have  been  named  in  a  manner  which  facilitates 
locating  a  label  in  the  source  listing. 

35.  The  machine  cross  reference  listings  appear  to  be  useful. 
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36.  This  module's  flow  chart  represents  the  logic  control  flow  as 

shown  in  this  module's  source  listing. 

37.  This  module's  flow  chart  represents  the  data  flow  as  shown  in 

this  module's  source  listing. 

38.  The  labels  in  this  module's  flow  chart  and  the  statement  labels 
in  this  module's  source  listing  are  in  agr-oomont. 

39.  The  inputs  to  this  module  as  described  in  the  documentation 

correspond  to  the  inputs  as  shown  in  this  iiodule's  source 

1 i sting. 

40.  The  outputs  from  this  module  as  described  in  the  documentation 

correspond  to  the  outputs  as  shown  in  the  module's  source 

1 i sti ng . 

41.  The  order  of  arguments  for  this  module  as  described  in  the  docu¬ 
mentation  corresponds  to  the  order  of  arguments  as  shown  in  this 
module's  source  listing. 

42.  The  module  processing  as  described  in  the  documentation 
corresponds  to  the  implemented  processing  as  shown  in  this 
module's  source  listing. 

43.  The  programming  conventions  established  in  the  documentation  for 
source  code  development  have  been  followed  within  this  module. 

44.  The  delineation  of  comments  is  uniform  within  sections  of  this 
modul e. 

45.  Each  variable  in  this  module  is  considered  to  be  of  one  (and  only 
one)  data  type  for  all  occurrences. 

46.  Each  variable  in  this  module  has  only  one  function. 

47.  Global  variables  are  distinguishable  from  local  variables  by  a 
naming  convention. 

48.  The  use  of  indentation  is  uniform  within  this  module. 

49.  The  information  in  the  preface  block  is  consistent  with  the  asso¬ 
ciated  source  code. 


SIMPLICITY  QUESTIONS 


50.  The  source  language  for  this  module  is  a  high  order  language 
(HOD. 


51.  The  control  flow  of  this  niodule  Is  essentially  from  top  to  bot¬ 
tom. 

6?,  This  module  contains  very  little  extraneous  code. 

53.  There  is  minimal  use  of  sficcialized  coding  techniques  in  this 
modul  c . 

54.  Esoteric  (clever)  programining  is  avoided  in  this  module. 

55.  GO  TO-"' ike  branch  statements  in  this  module  are  used  only  where 
essenti al . 

56.  There  is  reasonable  use  of  statement  labels  in  this  module, 

57.  A  knowledge  of  mathematics  beyond  basic  algebra  is  not  required 
to  understand  the  mathematical  functions  performied  by  this 
modul  e. 

58.  This  module  contains  a  minimal  number  of  compound  data  struc¬ 
tures  . 

59.  This  module  contains  a  minimal  number  of  compound  control  struc¬ 
tures  . 

60.  Each  physical  source  line  in  this  module  contains  at  most  one 
executable  source  statement. 

61.  There  is  a  minimal  use  of  compound  Boolean  expressions  in  this 
modul  e . 

62.  The  number  of  expressions  used  to  control  branching  in  this 
module  is  manageable. 

63.  The  number  of  unique  operators  in  this  module  is  manageable. 

64.  The  number  of  unique  operands  in  this  module  is  manageable. 

65.  The  number  of  executable  st.<^tements  in  this  (nodule  is  manageable. 

EXPANDABILITY  QUESTIONS 

66.  There  is  a  minimal  mixing  of  1/'''  functions  and  other  application 
functions  in  this  modulo. 

67.  There  is  a  minimal  mixing  of  machiue  dt^pendent  functions  and 
other  application  functions  in  this  module. 


f:«s.  Const.^nts  used  niore  th.Ki  once  in  this  'ood'jle  are  para  fieri  zed. 

There  is  minimal  use  of  process  i  ng-depefident  code  (e.g.,  relative 
addressing,  sel f-modi tying  code,  etc*)  in  this  module. 

/O.  The  size  of  any  data  structure  which  affects  the  processing  logic 
of  this  module  is  parameteri zed , 

'M  .  Any  constants  (e.g.,  accuracy,  convergence,  timing)  which  affect 
processing  in  this  mcdule  are  parameteri zed . 

^2.  The  contribution  of  this  module  to  the  cons;jm[>tion  of  frame  tine 
can  be  determined. 

M.  The  volume  of  data  wni^h  this  module  can  process  does  not  appear 
to  be  limited. 

-T.  It  appears  that  tijnrtional  parts  could  be  easily  inserted, 
deleted,  or  replaced  within  this  module. 


INoTRUMfNTATION  QUEST lOKS 

'^b.  This  module  contains  checks  for  possible  out-of-bound  array 
suhscriots . 

.  This  module  contains  nocks  to  detect  possible  unc'  fined  opera¬ 

tions  . 

•'7.  This  nodule  ^  •ninifna''  v'uiunt  of  '^ode  which  would  require 

1  owe r - 1  0 V el  U ('tail  eo  "  ^  i  n q  . 

-  Source  listing  soss.ie-  ♦  ungest  or  referer^Ci-  input  data  <ind  asso¬ 
ciated  oiitput  results  v  use  in  testing  tins  module. 

n  Diagnostic  ■’Messages  on'u  r  codes  are  output  when  an  illegal  input 
to  this  nifdule  is  encou^  tored. 

Diagnostic'  •  essages/error  codes  are  output  u  ,  t  an  internal 
module  fai  ure  could  occur. 

■•1.  Intermediae  results  within  this  module  can  he  ‘selectively 

col1ectc*i  or  display. 

Aids  exist  in  jr  ^  Of'  <  »  ily  into  the  module's  s^’u^'ie 

code  for  the  lu'^'oese  -  ■'•  i  ’  a-  logical  flow  of  conteo!  . 


GENERAL  QUESTIONS 


83.  Modularity  as  reflected  in  this  module's  source  listing  contrib¬ 
utes  to  the  maintainability  of  this  module. 

84.  Descriptiveness  as  reflected  in  this  module's  source  listing 

contributes  to  the  maintainability  of  this  module. 

85.  Consistency  as  reflected  in  this  .Jiodule's  source  listing  and  be¬ 
tween  the  source  listing  and*  documentation  contributes  to  the 
maintainability  of  this  module. 

86.  Simplicity  as  reflected  in  this  module's  source  listing  contrib¬ 
utes  to  the  maintainability  of  this  module. 

87.  Expandability  as  reflected  in  this  module's  source  listing 

contributes  to  the  maintainability  of  this  module. 

88.  Instrumentation  as  reflected  in  this  module's  source  listing 

contributes  to  the  maintainability  of  this  module. 

89.  Overall  it  appears  that  the  characteristics  of  this  module's 

source  listing  contribute  to  the  maintainability  of  this  module. 
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Sr^TWARF  OPFRATt^R-^'^ACHINE  iNTKRrA':;  ST  K^JNA  IRE 


ASa'IIRAPILITV  questions 

Assurabi  1  i ty :  the  extent  that  the  systern  aids  the  r; 
val^'da'tinp  data,  avoiding  errors,  and  correc t ran  errors  err.  < 

1.  Operator  input  errors  do  not  eause  system  iailurc^s. 

Operator  input  errors  are  detected. 

3.  "'he  causes  of  input  errors  are  displayed  to  thr 

A.  'he  action  renuired  to  <Ofrrct  an  op(‘’-5tc.r  ir-put 
;  i  s p  1  ay  e d  to  the  o pe  r  a  to  r  , 

;-'Put  errors  are  easily  cor'rected, 

c.  In  rut  errors  are  quickly  c  cu'rec  ted. 

The  operator  can  verify  input  before  ('xocutu>n  rntry. 

S.  ^Mssion  pecul  iar  data  entered  by  th(‘  opt  rato;r  is 
V  a^  idi tv  . 

n.  ih^a  data  entry  display  has  a  cursor  or  pointer. 

;  .  ■  ,.erj.tcr  s  able  tn  corrpct  mistyiHui  L^irrt, 

:  ac  ksi'ace  key . 

1..  '  docs  not  require  trie  operator  to  (  oi  ^  ’nfr 

'  .  r  ' ; . 
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to 

fof't  rol )  ahi  1 1  ty :  the  extent  'Ca 

.liroct  the  oporMtions  o*  the  nvi  r.' 

*  *  f  •  •.  V  s  t  eMl  al  1  ov> 

■  e  opera  tor 

w 

The  operate^'  can  interrojpt  v)  i  » 

Hit  ■  I'lati pa'O; 

e-  >es. 

)  c 

The  operator  has  task  ahoet  ci  0 

' '  '  1  1  C' s  a  *  -U  1  ah  1  0  . 

!  . 

^he  oiHVMtor  may  initiate  sel*'  r 

■  •  ,  s ,  s  ^ ■  ■  ■ 

■0 . 

The  operator'  may  send  injtpot 
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;'i . 

The  operator'  can  as^  for  .e''  : 
operation . 

'  •  t  1  VC  the  cur^'o' 

t  status  of 

??. 

The  operator-  can  t  tn^ 
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r  1  f  ri’'  1 1 1 
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d  3 . 
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*  a tus  . 

0  .'1 

The  0(10  r  a  tor'  jiiay  lortr'id  the  n 
both  in  [Hit  and  oiitput. 

*  .0  •  t'Oe  i  ra  d  pi 

riatory  text. 

The  oper'ator-  may  edit  tn<'  la» 

i  ■  ,  '  !  V  tc  ['V 

the  system. 

The  «>p^''^'dtor  may  Lrt'ate  an:  -  v 
single  c  nrmiand. 

•  *  i  :  ‘  1 "  ,  .  0 1 

-  rmands  as  a 

T he  0 [ >e r a  tor*  c a ni  c omm a n ( i  v  1  s  1 :  '  i 
matic  pr'oeosses . 

'  ■  V  t  opt  d  t  A  CV  ill 

V n )  of  auto- 

. 

The  operator  mav  command  dif^-  ’o 

v  des  of  opera  f  i  or  . 

, '  0  ^ 

The  operatt>r  may  Cfintr'ol  the  ^yp( 

arod  uuintity  of  output. 

.  Rypass  procrfluros  avdilaMf’  o  t  I  in  of  partial 

system  failiire  the  mor'e  import. r't  -.v  , t.vn  fijru  tion'  can  still  he 
per  former) . 

OA!)  Rf  AS(ihA’M!  1 1  Y  Ollt  M  UUJS 

Workload  msonihilHy:  the  extr-  *  tmit  thr-  tas^s  required  of 

tte  operator  aro*  within  the  operator  s  ^,1:  <ind  the  extent  to 

which  the  operator  pt^rforms  a  useful,  ’  ' 

Vi.  It  is  easy  to  entt^r  mission  (tasM  ;  li  m»'  data. 

Data  preparation  is  usiially  perf^*'  '  n  :  on-line  devices. 
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33.  The  system  will  accept  free-rormat  commands  and  data. 

34.  Menu  techniques  are  used  to  aid  the  operator  in  making  decisions. 


35.  The  system  may  be  operated  without  reference  to  manuals  during 

normal  operations. 

36.  The  operator  needs  to  memorize  a  comfortably  small  number  of  Lom- 
mands  in  order  to  effectively  operate  the  system. 

37.  Messages  to  the  operator  a  "e  easy  to  understand. 

38.  The  device  used  to  send  messages  to  the  operator  provides  info'^- 

mation  at  a  rate  comfortable  to  the  operator, 

39.  The  number  of  messages  presented  to  the  operator  at  one  time  is 
smal 1  . 

40.  The  system  software  may  be  reloaded  quickly  and  easily. 

41.  The  system  software  needs  to  be  reloaded  ir frequently . 

42.  System  warm-up  time  is  small. 

43.  The  operator's  manual  makes  minimal  use  of  cross-references. 

44.  It  is  easy  to  locate  specific  information  within  the  operator's 
manual  . 

45.  The  operator's  manual  is  a  reasonable  size. 

4b.  The  operator  performs  no  tedious  functions  which  could  be  handled 

by  the  system. 

4'.  The  operator  is  rarely  bored  and  performs  a  "dynamic"  function. 

48.  The  operator  is  not  forced  to  wait  for  the  machine  to  respond. 

4^.  The  tor  is  not  a  slave  to  the  machine. 
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ne^(  r  i  pti  veness :  the  cxtfv  t  to  which  tfie  operator  has  available 

'i*  Vjiied  px pi  ana t ions  of  vvrry  fuOLtion  operator  perfrnps  and 

the  machine  perfijnas. 

‘  0.  C;  V.*  r-on  and  power-off  |  pom  are  wrli  documented. 


51.  The  operator  has  adequate  instructions  for  handling  emergencies. 

52.  Legitimate  responses  for  all  conditions  are  explained. 

53.  The  software  provides  a  question-answer  type  operator  aid. 

54.  The  system  will  explain  each  command  upon  user  request. 

55.  Explanations  of  how  to  interpret  all  output  data  are  available. 

56.  The  operator  is  adequately  alerted  when  the  system  requires 
operator  action. 

57.  The  machine  gives  the  operator  decision  aids  if  tasks  cannot  be 
executed  as  ordered. 

58.  The  version  number  (revision  number)  of  the  software  is  readily 
available  to  the  operator  from  the  system. 

59.  Data  base  configuration  data  are  readily  available  to  the  opera¬ 
tor. 

60.  All  documents  the  operator  requires  (including  cross-references) 
are  easily  available  to  him. 

61.  The  operator's  manual  clearly  explains  the  normal  sequential 
steps  of  operation. 

62.  The  operator's  manual  contains  a  useful  table  of  contents. 

63.  The  operator's  manual  contains  a  useful  index. 

64.  The  operator's  manual  contains  a  useful  glossary. 

C0MSISTE^jCY  QUESTIONS 

r onsi stency :  the  extent  that  the  behavior  of  the  machine  and 

docunuTitation  correspond  to  the  expectations  of  the  operator. 

65.  Operator  entered  commands  are  systematical  ly  forinatted. 

66.  The  command  langi...ge  is  a  standardized  language. 

67.  Requirements  for  operator  input  agree  with  the  operator's  manual . 

68.  Messages  to  the  operator  are  systematical  ly  formatted. 


69.  Messages  requiring  action  by  the  operator  are  always  highlighted 
in  some  fashion. 

70.  Operator  entries  always  result  in  some  type  of  response. 

71.  Response  times  are  similar  for  groups  of  similar  activities. 

72.  System  performance  corresponds  with  documented  performance 
(specifications,  user's  manuals,  etc.). 

73.  Checklists  agree  with  the  operator's  manual. 

74.  Operator's  manuals  are  systematically  formatted. 

SIMPLICITY  QUESTIONS 

Simp! i ci ty :  the  extent  that  information  presented  to  the  opera¬ 
tor  or  entered  by  the  operator  is  grouped  into  short,  readily 
understandabl  e  structures. 

75.  The  operator  needs  to  know  only  one  command  language. 

76.  Operator  entered  instructions  are  relatively  short. 

77.  It  is  easy  to  understand  actions  reqiiired  of  the  operator. 

78.  Messages  to  the  operator  are  short. 

70.  Each  new  message  contains  only  one  idea  to  which  the  operator 
must  respond. 

80.  Only  essential  or  useful  infomiation  is  displayed  to  the  opera¬ 
tor. 

81.  The  display  is  not  overcrowded  (unless  commanded  to  be  so). 

82.  Difficult  words  or  characters  are  rarely  used. 

83.  Data  structures  are  easily  understandable. 

84.  The  operator  has  appropriate  checklists  available. 

85.  The  number  of  checklists  required  is  manageable. 

86.  The  operator's  manual  is  a  single  volume  (except  for  checklists). 

87.  The  operator's  manual  is  easy  to  understand. 
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88.  Alternatives  to  normal  operating  sequences  are  described  sepa¬ 
rately  (not  embedded  within  normal  procedures). 


GENERAL  QIILSTIONS 

Note:  The  following  questions  relate  to  the  evaluator's  general 

impression  of  the  computer  program's  contribution  to  system  usability 
or  effectiveness.  Definitions  of  the  test  factors  should  be  reviewed 
before  comjiletinq  these  questions. 


89. 

The  concepts  of  Assurability  as  implemented 
contribute  to  usability  of  the  system. 

in  the 

system 

90. 

The  concepts  of  Control  1  abil  i ty  as  implemented 
contribute  to  usability  of  the  system. 

in  the  system 

91. 

The  concepts  of  Workload  Reasonabil i ty  as  implemented 
system  contribute  to  usability  of  the  system. 

i  n  the 

92. 

The  concepts  of  Descriptiveness  as  implemented 
contribute  to  usability  of  the  system. 

in  the  system 

93. 

The  concepts  of  Consistency  as  imnlemented  in  the 
ute  to  usability  of  the  system. 

system 

contr i b- 

94. 

^he  concepts  of  Simplicity  as  implemented  in  the 
lite  to  usability  of  the  system. 

system 

con tr 1 b> 

95. 

Overall,  it  appears  that  the  operaton-inachi  ne  interface 
well  'lesidoed. 

has  been 

iius  r.rviRNilfWT  Pdinr  «G  O' rc( 


